Sep 03 14:34:39 oak-md1-s2 kernel: microcode: microcode updated early to revision 0x3d, date = 2018-04-20 Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys cpuset Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys cpu Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys cpuacct Sep 03 14:34:39 oak-md1-s2 kernel: Linux version 3.10.0-693.2.2.el7_lustre.pl1.x86_64 (sthiell@oak-rbh01) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Sat Sep 30 00:44:25 PDT 2017 Sep 03 14:34:39 oak-md1-s2 kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.2.2.el7_lustre.pl1.x86_64 root=UUID=5bea4979-eae3-4e4b-a662-939c7a375303 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Sep 03 14:34:39 oak-md1-s2 kernel: e820: BIOS-provided physical RAM map: Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x0000000000100000-0x0000000067b5bfff] usable Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x0000000067b5c000-0x000000006fb63fff] reserved Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000006fb64000-0x000000007a288fff] usable Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000007a289000-0x000000007af0afff] reserved Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000007af0b000-0x000000007b93afff] ACPI NVS Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000007b93b000-0x000000007bab7fff] ACPI data Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000007bab8000-0x000000007bae8fff] usable Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000007bae9000-0x000000007bafefff] ACPI data Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000007baff000-0x000000007bafffff] usable Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x000000007bb00000-0x000000008fffffff] reserved Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x00000000feda8000-0x00000000fedabfff] reserved Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x00000000ff310000-0x00000000ffffffff] reserved Sep 03 14:34:39 oak-md1-s2 kernel: BIOS-e820: [mem 0x0000000100000000-0x000000207fffffff] usable Sep 03 14:34:39 oak-md1-s2 kernel: NX (Execute Disable) protection: active Sep 03 14:34:39 oak-md1-s2 kernel: SMBIOS 2.8 present. Sep 03 14:34:39 oak-md1-s2 kernel: DMI: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.6.0 10/26/2017 Sep 03 14:34:39 oak-md1-s2 kernel: e820: update [mem 0x00000000-0x00000fff] usable ==> reserved Sep 03 14:34:39 oak-md1-s2 kernel: e820: remove [mem 0x000a0000-0x000fffff] usable Sep 03 14:34:39 oak-md1-s2 kernel: e820: last_pfn = 0x2080000 max_arch_pfn = 0x400000000 Sep 03 14:34:39 oak-md1-s2 kernel: MTRR default type: uncachable Sep 03 14:34:39 oak-md1-s2 kernel: MTRR fixed ranges enabled: Sep 03 14:34:39 oak-md1-s2 kernel: 00000-9FFFF write-back Sep 03 14:34:39 oak-md1-s2 kernel: A0000-BFFFF uncachable Sep 03 14:34:39 oak-md1-s2 kernel: C0000-FFFFF write-protect Sep 03 14:34:39 oak-md1-s2 kernel: MTRR variable ranges enabled: Sep 03 14:34:39 oak-md1-s2 kernel: 0 base 000000000000 mask 3FE000000000 write-back Sep 03 14:34:39 oak-md1-s2 kernel: 1 base 002000000000 mask 3FFF80000000 write-back Sep 03 14:34:39 oak-md1-s2 kernel: 2 base 000080000000 mask 3FFF80000000 uncachable Sep 03 14:34:39 oak-md1-s2 kernel: 3 base 0000FF000000 mask 3FFFFF000000 write-protect Sep 03 14:34:39 oak-md1-s2 kernel: 4 disabled Sep 03 14:34:39 oak-md1-s2 kernel: 5 disabled Sep 03 14:34:39 oak-md1-s2 kernel: 6 disabled Sep 03 14:34:39 oak-md1-s2 kernel: 7 disabled Sep 03 14:34:39 oak-md1-s2 kernel: 8 disabled Sep 03 14:34:39 oak-md1-s2 kernel: 9 disabled Sep 03 14:34:39 oak-md1-s2 kernel: x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 Sep 03 14:34:39 oak-md1-s2 kernel: x2apic enabled by BIOS, switching to x2apic ops Sep 03 14:34:39 oak-md1-s2 kernel: e820: last_pfn = 0x7bb00 max_arch_pfn = 0x400000000 Sep 03 14:34:39 oak-md1-s2 kernel: Base memory trampoline at [ffff880000096000] 96000 size 24576 Sep 03 14:34:39 oak-md1-s2 kernel: Using GB pages for direct mapping Sep 03 14:34:39 oak-md1-s2 kernel: BRK [0x01fe9000, 0x01fe9fff] PGTABLE Sep 03 14:34:39 oak-md1-s2 kernel: BRK [0x01fea000, 0x01feafff] PGTABLE Sep 03 14:34:39 oak-md1-s2 kernel: BRK [0x01feb000, 0x01febfff] PGTABLE Sep 03 14:34:39 oak-md1-s2 kernel: BRK [0x01fec000, 0x01fecfff] PGTABLE Sep 03 14:34:39 oak-md1-s2 kernel: BRK [0x01fed000, 0x01fedfff] PGTABLE Sep 03 14:34:39 oak-md1-s2 kernel: BRK [0x01fee000, 0x01feefff] PGTABLE Sep 03 14:34:39 oak-md1-s2 kernel: RAMDISK: [mem 0x3516b000-0x368adfff] Sep 03 14:34:39 oak-md1-s2 kernel: Early table checksum verification disabled Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: RSDP 00000000000fe320 00024 (v02 DELL ) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: XSDT 000000007bab60e8 000C4 (v01 DELL PE_SC3 00000000 01000013) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: FACP 000000007bab2000 000F4 (v04 DELL PE_SC3 00000000 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: DSDT 000000007ba98000 10777 (v02 DELL PE_SC3 00000003 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: FACS 000000007b8d3000 00040 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: MCEJ 000000007bab5000 00130 (v01 INTEL 00000001 INTL 0100000D) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: WD__ 000000007bab4000 00134 (v01 DELL PE_SC3 00000001 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: SLIC 000000007bab3000 00024 (v01 DELL PE_SC3 00000001 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: HPET 000000007bab1000 00038 (v01 DELL PE_SC3 00000001 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: APIC 000000007baaf000 01FFC (v03 DELL PE_SC3 00000000 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: MCFG 000000007baae000 0003C (v01 DELL PE_SC3 00000001 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: MSCT 000000007baad000 00090 (v01 DELL PE_SC3 00000001 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: SLIT 000000007baac000 0006C (v01 DELL PE_SC3 00000001 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: SRAT 000000007baa9000 02330 (v03 DELL PE_SC3 00000002 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: SSDT 000000007b955000 1424A9 (v02 DELL PE_SC3 00004000 INTL 20121114) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: SSDT 000000007b952000 0217F (v02 DELL PE_SC3 00000002 INTL 20121114) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: SSDT 000000007b951000 0006E (v02 DELL PE_SC3 00000002 INTL 20121114) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PRAD 000000007b950000 00132 (v02 DELL PE_SC3 00000002 INTL 20121114) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: SPCR 000000007bafe000 00050 (v01 00000000 00000000) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: DMAR 000000007bafd000 00120 (v01 DELL PE_SC3 00000001 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: HEST 000000007baf1000 0017C (v01 DELL PE_SC3 00000002 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: BERT 000000007baf0000 00030 (v01 DELL PE_SC3 00000002 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: ERST 000000007baef000 00230 (v01 DELL PE_SC3 00000002 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: EINJ 000000007baee000 00150 (v01 DELL PE_SC3 00000002 DELL 00000001) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Local APIC address 0xfee00000 Sep 03 14:34:39 oak-md1-s2 kernel: Setting APIC routing to cluster x2apic. Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x00 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x10 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x02 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x12 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x04 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x14 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x06 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x16 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x08 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x18 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x0a -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x1a -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x01 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x11 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x03 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x13 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x05 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x15 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x07 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x17 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x09 -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x19 -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 0 -> APIC 0x0b -> Node 0 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: PXM 1 -> APIC 0x1b -> Node 1 Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: Node 0 PXM 0 [mem 0x00000000-0x107fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: NUMA: Initialized distance table, cnt=2 Sep 03 14:34:39 oak-md1-s2 kernel: NODE_DATA(0) allocated [mem 0x107ffd9000-0x107fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: NODE_DATA(1) allocated [mem 0x207ffd6000-0x207fffcfff] Sep 03 14:34:39 oak-md1-s2 kernel: Reserving 168MB of memory at 672MB for crashkernel (System RAM: 130850MB) Sep 03 14:34:39 oak-md1-s2 kernel: Zone ranges: Sep 03 14:34:39 oak-md1-s2 kernel: DMA [mem 0x00001000-0x00ffffff] Sep 03 14:34:39 oak-md1-s2 kernel: DMA32 [mem 0x01000000-0xffffffff] Sep 03 14:34:39 oak-md1-s2 kernel: Normal [mem 0x100000000-0x207fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: Movable zone start for each node Sep 03 14:34:39 oak-md1-s2 kernel: Early memory node ranges Sep 03 14:34:39 oak-md1-s2 kernel: node 0: [mem 0x00001000-0x0009bfff] Sep 03 14:34:39 oak-md1-s2 kernel: node 0: [mem 0x00100000-0x67b5bfff] Sep 03 14:34:39 oak-md1-s2 kernel: node 0: [mem 0x6fb64000-0x7a288fff] Sep 03 14:34:39 oak-md1-s2 kernel: node 0: [mem 0x7bab8000-0x7bae8fff] Sep 03 14:34:39 oak-md1-s2 kernel: node 0: [mem 0x7baff000-0x7bafffff] Sep 03 14:34:39 oak-md1-s2 kernel: node 0: [mem 0x100000000-0x107fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: node 1: [mem 0x1080000000-0x207fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: Initmem setup node 0 [mem 0x00001000-0x107fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: On node 0 totalpages: 16720462 Sep 03 14:34:39 oak-md1-s2 kernel: DMA zone: 64 pages used for memmap Sep 03 14:34:39 oak-md1-s2 kernel: DMA zone: 21 pages reserved Sep 03 14:34:39 oak-md1-s2 kernel: DMA zone: 3995 pages, LIFO batch:0 Sep 03 14:34:39 oak-md1-s2 kernel: DMA32 zone: 7243 pages used for memmap Sep 03 14:34:39 oak-md1-s2 kernel: DMA32 zone: 463539 pages, LIFO batch:31 Sep 03 14:34:39 oak-md1-s2 kernel: Normal zone: 253952 pages used for memmap Sep 03 14:34:39 oak-md1-s2 kernel: Normal zone: 16252928 pages, LIFO batch:31 Sep 03 14:34:39 oak-md1-s2 kernel: Initmem setup node 1 [mem 0x1080000000-0x207fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: On node 1 totalpages: 16777216 Sep 03 14:34:39 oak-md1-s2 kernel: Normal zone: 262144 pages used for memmap Sep 03 14:34:39 oak-md1-s2 kernel: Normal zone: 16777216 pages, LIFO batch:31 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PM-Timer IO Port: 0x408 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Local APIC address 0xfee00000 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0x10] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0x12] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0x14] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0x16] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0x18] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x1a] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0x11] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0x13] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0x15] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0x17] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0x19] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x1b] enabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x00] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x01] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x02] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x03] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x04] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x05] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x06] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x07] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x08] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x09] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x0a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x0b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x0c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x0d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x0e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x0f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x10] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x11] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x12] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x13] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x14] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x15] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x16] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x17] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x18] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x19] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x1a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x1b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x1c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x1d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x1e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x1f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x20] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x21] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x22] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x23] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x24] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x25] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x26] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x27] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x28] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x29] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x2a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x2b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x2c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x2d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x2e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x2f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x30] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x31] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x32] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x33] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x34] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x35] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x36] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x37] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x38] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x39] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x3a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x3b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x3c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x3d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x3e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x3f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x40] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x41] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x42] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x43] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x44] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x45] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x46] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x47] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x48] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x49] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x4a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x4b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x4c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x4d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x4e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x4f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x50] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x51] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x52] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x53] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x54] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x55] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x56] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x57] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x58] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x59] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x5a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x5b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x5c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x5d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x5e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x5f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x60] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x61] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x62] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x63] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x64] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x65] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x66] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x67] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x68] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x69] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x6a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x6b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x6c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x6d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x6e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x6f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x70] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x71] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x72] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x73] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x74] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x75] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x76] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x77] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x78] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x79] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x7a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x7b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x7c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x7d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x7e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x7f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x80] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x81] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x82] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x83] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x84] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x85] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x86] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x87] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x88] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x89] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x8a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x8b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x8c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x8d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x8e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x8f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x90] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x91] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x92] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x93] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x94] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x95] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x96] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x97] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x98] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x99] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x9a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x9b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x9c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x9d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x9e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0x9f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa0] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa1] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa2] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa3] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa4] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa5] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa6] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa7] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa8] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xa9] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xaa] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xab] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xac] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xad] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xae] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xaf] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb0] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb1] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb2] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb3] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb4] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb5] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb6] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb7] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb8] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xb9] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xba] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xbb] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xbc] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xbd] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xbe] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: X2APIC_NMI (uid[0xbf] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x18] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x19] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x1a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x1b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x1c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x1d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x1e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x1f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x20] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x21] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x22] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x23] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x24] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x25] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x26] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x27] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x28] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x29] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x2a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x2b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x2c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x2d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x2e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x2f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x30] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x31] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x32] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x33] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x34] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x35] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x36] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x37] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x38] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x39] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x3a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x3b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x3c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x3d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x3e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x3f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x40] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x41] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x42] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x43] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x44] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x45] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x46] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x47] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x48] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x49] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x4a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x4b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x4c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x4d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x4e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x4f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x50] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x51] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x52] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x53] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x54] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x55] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x56] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x57] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x58] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x59] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x5a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x5b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x5c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x5d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x5e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x5f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x60] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x61] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x62] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x63] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x64] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x65] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x66] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x67] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x68] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x69] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x6a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x6b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x6c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x6d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x6e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x6f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x70] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x71] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x72] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x73] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x74] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x75] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x76] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x77] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x78] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x79] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x7a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x7b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x7c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x7d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x7e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x7f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x80] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x81] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x82] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x83] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x84] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x85] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x86] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x87] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x88] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x89] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x8a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x8b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x8c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x8d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x8e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x8f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x90] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x91] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x92] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x93] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x94] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x95] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x96] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x97] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x98] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x99] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x9a] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x9b] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x9c] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x9d] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x9e] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0x9f] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa0] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa1] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa2] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa3] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa4] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa5] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa6] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa7] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa8] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xa9] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xaa] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xab] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xac] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xad] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xae] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xaf] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb0] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb1] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb2] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb3] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb4] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb5] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb6] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb7] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb8] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xb9] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xba] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xbb] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xbc] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xbd] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xbe] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: LAPIC_NMI (acpi_id[0xbf] high level lint[0x1]) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) Sep 03 14:34:39 oak-md1-s2 kernel: IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24]) Sep 03 14:34:39 oak-md1-s2 kernel: IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48]) Sep 03 14:34:39 oak-md1-s2 kernel: IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: IRQ0 used by override. Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: IRQ9 used by override. Sep 03 14:34:39 oak-md1-s2 kernel: Using ACPI (MADT) for SMP configuration information Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: HPET id: 0x8086a701 base: 0xfed00000 Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Allowing 384 CPUs, 360 hotplug CPUs Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x0009c000-0x0009ffff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x67b5c000-0x6fb63fff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x7a289000-0x7af0afff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x7af0b000-0x7b93afff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x7b93b000-0x7bab7fff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x7bae9000-0x7bafefff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x7bb00000-0x8fffffff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0x90000000-0xfeda7fff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0xfeda8000-0xfedabfff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0xfedac000-0xff30ffff] Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registered nosave memory: [mem 0xff310000-0xffffffff] Sep 03 14:34:39 oak-md1-s2 kernel: e820: [mem 0x90000000-0xfeda7fff] available for PCI devices Sep 03 14:34:39 oak-md1-s2 kernel: Booting paravirtualized kernel on bare hardware Sep 03 14:34:39 oak-md1-s2 kernel: setup_percpu: NR_CPUS:5120 nr_cpumask_bits:384 nr_cpu_ids:384 nr_node_ids:2 Sep 03 14:34:39 oak-md1-s2 kernel: PERCPU: Embedded 33 pages/cpu @ffff88103ce00000 s97048 r8192 d29928 u262144 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: s97048 r8192 d29928 u262144 alloc=1*2097152 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 000 002 004 006 008 010 012 014 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 016 018 020 022 024 026 028 030 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 032 034 036 038 040 042 044 046 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 048 050 052 054 056 058 060 062 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 064 066 068 070 072 074 076 078 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 080 082 084 086 088 090 092 094 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 096 098 100 102 104 106 108 110 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 112 114 116 118 120 122 124 126 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 128 130 132 134 136 138 140 142 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 144 146 148 150 152 154 156 158 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 160 162 164 166 168 170 172 174 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 176 178 180 182 184 186 188 190 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 192 194 196 198 200 202 204 206 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 208 210 212 214 216 218 220 222 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 224 226 228 230 232 234 236 238 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 240 242 244 246 248 250 252 254 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 256 258 260 262 264 266 268 270 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 272 274 276 278 280 282 284 286 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 288 290 292 294 296 298 300 302 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 304 306 308 310 312 314 316 318 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 320 322 324 326 328 330 332 334 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 336 338 340 342 344 346 348 350 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 352 354 356 358 360 362 364 366 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [0] 368 370 372 374 376 378 380 382 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 001 003 005 007 009 011 013 015 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 017 019 021 023 025 027 029 031 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 033 035 037 039 041 043 045 047 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 049 051 053 055 057 059 061 063 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 065 067 069 071 073 075 077 079 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 081 083 085 087 089 091 093 095 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 097 099 101 103 105 107 109 111 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 113 115 117 119 121 123 125 127 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 129 131 133 135 137 139 141 143 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 145 147 149 151 153 155 157 159 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 161 163 165 167 169 171 173 175 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 177 179 181 183 185 187 189 191 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 193 195 197 199 201 203 205 207 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 209 211 213 215 217 219 221 223 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 225 227 229 231 233 235 237 239 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 241 243 245 247 249 251 253 255 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 257 259 261 263 265 267 269 271 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 273 275 277 279 281 283 285 287 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 289 291 293 295 297 299 301 303 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 305 307 309 311 313 315 317 319 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 321 323 325 327 329 331 333 335 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 337 339 341 343 345 347 349 351 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 353 355 357 359 361 363 365 367 Sep 03 14:34:39 oak-md1-s2 kernel: pcpu-alloc: [1] 369 371 373 375 377 379 381 383 Sep 03 14:34:39 oak-md1-s2 kernel: Built 2 zonelists in Zone order, mobility grouping on. Total pages: 32974254 Sep 03 14:34:39 oak-md1-s2 kernel: Policy zone: Normal Sep 03 14:34:39 oak-md1-s2 kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.2.2.el7_lustre.pl1.x86_64 root=UUID=5bea4979-eae3-4e4b-a662-939c7a375303 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 Sep 03 14:34:39 oak-md1-s2 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 Sep 03 14:34:39 oak-md1-s2 kernel: xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form Sep 03 14:34:39 oak-md1-s2 kernel: Memory: 5786596k/136314880k available (6886k kernel code, 2324168k absent, 2426916k reserved, 4545k data, 1764k init) Sep 03 14:34:39 oak-md1-s2 kernel: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=384, Nodes=2 Sep 03 14:34:39 oak-md1-s2 kernel: Hierarchical RCU implementation. Sep 03 14:34:39 oak-md1-s2 kernel: RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=384. Sep 03 14:34:39 oak-md1-s2 kernel: NR_IRQS:327936 nr_irqs:4312 0 Sep 03 14:34:39 oak-md1-s2 kernel: Console: colour VGA+ 80x25 Sep 03 14:34:39 oak-md1-s2 kernel: console [ttyS0] enabled Sep 03 14:34:39 oak-md1-s2 kernel: allocated 536870912 bytes of page_cgroup Sep 03 14:34:39 oak-md1-s2 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Sep 03 14:34:39 oak-md1-s2 kernel: Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl Sep 03 14:34:39 oak-md1-s2 kernel: hpet clockevent registered Sep 03 14:34:39 oak-md1-s2 kernel: tsc: Fast TSC calibration using PIT Sep 03 14:34:39 oak-md1-s2 kernel: spurious 8259A interrupt: IRQ7. Sep 03 14:34:39 oak-md1-s2 kernel: tsc: Detected 3399.763 MHz processor Sep 03 14:34:39 oak-md1-s2 kernel: Calibrating delay loop (skipped), value calculated using timer frequency.. 6799.52 BogoMIPS (lpj=3399763) Sep 03 14:34:39 oak-md1-s2 kernel: pid_max: default: 393216 minimum: 3072 Sep 03 14:34:39 oak-md1-s2 kernel: Security Framework initialized Sep 03 14:34:39 oak-md1-s2 kernel: SELinux: Initializing. Sep 03 14:34:39 oak-md1-s2 kernel: SELinux: Starting in permissive mode Sep 03 14:34:39 oak-md1-s2 kernel: Yama: becoming mindful. Sep 03 14:34:39 oak-md1-s2 kernel: Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: Mount-cache hash table entries: 262144 (order: 9, 2097152 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: Mountpoint-cache hash table entries: 262144 (order: 9, 2097152 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys memory Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys devices Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys freezer Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys net_cls Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys blkio Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys perf_event Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys hugetlb Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys pids Sep 03 14:34:39 oak-md1-s2 kernel: Initializing cgroup subsys net_prio Sep 03 14:34:39 oak-md1-s2 kernel: CPU: Physical Processor ID: 0 Sep 03 14:34:39 oak-md1-s2 kernel: CPU: Processor Core ID: 0 Sep 03 14:34:39 oak-md1-s2 kernel: ENERGY_PERF_BIAS: Set to 'normal', was 'performance' Sep 03 14:34:39 oak-md1-s2 kernel: ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) Sep 03 14:34:39 oak-md1-s2 kernel: mce: CPU supports 22 MCE banks Sep 03 14:34:39 oak-md1-s2 kernel: CPU0: Thermal monitoring enabled (TM1) Sep 03 14:34:39 oak-md1-s2 kernel: Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 Sep 03 14:34:39 oak-md1-s2 kernel: Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0 Sep 03 14:34:39 oak-md1-s2 kernel: tlb_flushall_shift: 6 Sep 03 14:34:39 oak-md1-s2 kernel: Freeing SMP alternatives: 24k freed Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Core revision 20130517 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: All ACPI Tables successfully acquired Sep 03 14:34:39 oak-md1-s2 kernel: ftrace: allocating 26589 entries in 104 pages Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Max logical packages: 64 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: Host address width 46 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: RMRR base: 0x00000067b5c000 end: 0x0000006fb63fff Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: RMRR base: 0x0000007ae07000 end: 0x0000007af06fff Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: ATSR flags: 0x0 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: ATSR flags: 0x0 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: Failed to copy IR table for dmar0 from previous kernel Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: Failed to copy IR table for dmar1 from previous kernel Sep 03 14:34:39 oak-md1-s2 kernel: DMAR-IR: Enabled IRQ remapping in x2apic mode Sep 03 14:34:39 oak-md1-s2 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (fam: 06, model: 3f, stepping: 02) Sep 03 14:34:39 oak-md1-s2 kernel: TSC deadline timer enabled Sep 03 14:34:39 oak-md1-s2 kernel: Performance Events: PEBS fmt2+, 16-deep LBR, Haswell events, full-width counters, Intel PMU driver. Sep 03 14:34:39 oak-md1-s2 kernel: ... version: 3 Sep 03 14:34:39 oak-md1-s2 kernel: ... bit width: 48 Sep 03 14:34:39 oak-md1-s2 kernel: ... generic registers: 4 Sep 03 14:34:39 oak-md1-s2 kernel: ... value mask: 0000ffffffffffff Sep 03 14:34:39 oak-md1-s2 kernel: ... max period: 00007fffffffffff Sep 03 14:34:39 oak-md1-s2 kernel: ... fixed-purpose events: 3 Sep 03 14:34:39 oak-md1-s2 kernel: ... event mask: 000000070000000f Sep 03 14:34:39 oak-md1-s2 kernel: NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #1 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #2 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #3 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #4 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #5 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #6 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #7 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #8 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #9 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #10 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #11 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #12 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #13 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #14 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #15 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #16 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #17 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #18 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #19 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #20 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #21 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 0, Processors #22 OK Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Booting Node 1, Processors #23 Sep 03 14:34:39 oak-md1-s2 kernel: Brought up 24 CPUs Sep 03 14:34:39 oak-md1-s2 kernel: smpboot: Total of 24 processors activated (163241.06 BogoMIPS) Sep 03 14:34:39 oak-md1-s2 kernel: node 0 initialised, 15460036 pages in 194ms Sep 03 14:34:39 oak-md1-s2 kernel: node 1 initialised, 15984260 pages in 196ms Sep 03 14:34:39 oak-md1-s2 kernel: devtmpfs: initialized Sep 03 14:34:39 oak-md1-s2 kernel: EVM: security.selinux Sep 03 14:34:39 oak-md1-s2 kernel: EVM: security.ima Sep 03 14:34:39 oak-md1-s2 kernel: EVM: security.capability Sep 03 14:34:39 oak-md1-s2 kernel: PM: Registering ACPI NVS region [mem 0x7af0b000-0x7b93afff] (10682368 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: atomic64 test passed for x86-64 platform with CX8 and with SSE Sep 03 14:34:39 oak-md1-s2 kernel: pinctrl core: initialized pinctrl subsystem Sep 03 14:34:39 oak-md1-s2 kernel: RTC time: 21:34:34, date: 09/03/18 Sep 03 14:34:39 oak-md1-s2 kernel: NET: Registered protocol family 16 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI FADT declares the system doesn't support PCIe ASPM, so disable it Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: bus type PCI registered Sep 03 14:34:39 oak-md1-s2 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Sep 03 14:34:39 oak-md1-s2 kernel: PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) Sep 03 14:34:39 oak-md1-s2 kernel: PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 Sep 03 14:34:39 oak-md1-s2 kernel: pmd_set_huge: Cannot satisfy [mem 0x80000000-0x80200000] with a huge-page mapping due to MTRR override. Sep 03 14:34:39 oak-md1-s2 kernel: PCI: Using configuration type 1 for base access Sep 03 14:34:39 oak-md1-s2 kernel: PCI: Dell System detected, enabling pci=bfsort. Sep 03 14:34:39 oak-md1-s2 kernel: core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Added _OSI(Module Device) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Added _OSI(Processor Device) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Added _OSI(3.0 _SCP Extensions) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Added _OSI(Processor Aggregator Device) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: EC: Look up EC in DSDT Sep 03 14:34:39 oak-md1-s2 kernel: [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Interpreter enabled Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: (supports S0 S5) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Using IOAPIC for interrupt routing Sep 03 14:34:39 oak-md1-s2 kernel: HEST: Table parsing has been initialized. Sep 03 14:34:39 oak-md1-s2 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff]) Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:02: PCIe AER handled by firmware Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration Sep 03 14:34:39 oak-md1-s2 kernel: PCI host bridge to bus 0000:ff Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:ff: root bus resource [bus ff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:08.0: [8086:2f80] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:08.2: [8086:2f32] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:08.3: [8086:2f83] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:09.0: [8086:2f90] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:09.2: [8086:2f33] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:09.3: [8086:2f93] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0b.0: [8086:2f81] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0b.1: [8086:2f36] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0b.2: [8086:2f37] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.0: [8086:2fe0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.1: [8086:2fe1] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.2: [8086:2fe2] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.3: [8086:2fe3] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.4: [8086:2fe4] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.5: [8086:2fe5] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.6: [8086:2fe6] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0c.7: [8086:2fe7] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0f.0: [8086:2ff8] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0f.1: [8086:2ff9] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0f.4: [8086:2ffc] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0f.5: [8086:2ffd] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:0f.6: [8086:2ffe] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:10.0: [8086:2f1d] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:10.1: [8086:2f34] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:10.5: [8086:2f1e] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:10.6: [8086:2f7d] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:10.7: [8086:2f1f] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:12.0: [8086:2fa0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:12.1: [8086:2f30] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:12.2: [8086:2f70] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.0: [8086:2fa8] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.1: [8086:2f71] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.2: [8086:2faa] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.3: [8086:2fab] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.4: [8086:2fac] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.5: [8086:2fad] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.6: [8086:2fae] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:13.7: [8086:2faf] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.0: [8086:2fb0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.1: [8086:2fb1] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.2: [8086:2fb2] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.3: [8086:2fb3] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.4: [8086:2fbc] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.5: [8086:2fbd] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.6: [8086:2fbe] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:14.7: [8086:2fbf] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:15.0: [8086:2fb4] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:15.1: [8086:2fb5] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:15.2: [8086:2fb6] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:15.3: [8086:2fb7] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:16.0: [8086:2f68] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:16.6: [8086:2f6e] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:16.7: [8086:2f6f] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:17.0: [8086:2fd0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:17.4: [8086:2fb8] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:17.5: [8086:2fb9] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:17.6: [8086:2fba] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:17.7: [8086:2fbb] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:1e.0: [8086:2f98] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:1e.1: [8086:2f99] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:1e.2: [8086:2f9a] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:1e.3: [8086:2fc0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:1e.4: [8086:2f9c] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:1f.0: [8086:2f88] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:ff:1f.2: [8086:2f8a] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f]) Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:03: PCIe AER handled by firmware Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration Sep 03 14:34:39 oak-md1-s2 kernel: PCI host bridge to bus 0000:7f Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:7f: root bus resource [bus 7f] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:08.0: [8086:2f80] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:08.2: [8086:2f32] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:08.3: [8086:2f83] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:09.0: [8086:2f90] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:09.2: [8086:2f33] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:09.3: [8086:2f93] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0b.0: [8086:2f81] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0b.1: [8086:2f36] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0b.2: [8086:2f37] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.0: [8086:2fe0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.1: [8086:2fe1] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.2: [8086:2fe2] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.3: [8086:2fe3] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.4: [8086:2fe4] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.5: [8086:2fe5] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.6: [8086:2fe6] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0c.7: [8086:2fe7] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0f.0: [8086:2ff8] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0f.1: [8086:2ff9] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0f.4: [8086:2ffc] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0f.5: [8086:2ffd] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:0f.6: [8086:2ffe] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:10.0: [8086:2f1d] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:10.1: [8086:2f34] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:10.5: [8086:2f1e] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:10.6: [8086:2f7d] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:10.7: [8086:2f1f] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:12.0: [8086:2fa0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:12.1: [8086:2f30] type 00 class 0x110100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:12.2: [8086:2f70] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.0: [8086:2fa8] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.1: [8086:2f71] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.2: [8086:2faa] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.3: [8086:2fab] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.4: [8086:2fac] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.5: [8086:2fad] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.6: [8086:2fae] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:13.7: [8086:2faf] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.0: [8086:2fb0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.1: [8086:2fb1] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.2: [8086:2fb2] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.3: [8086:2fb3] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.4: [8086:2fbc] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.5: [8086:2fbd] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.6: [8086:2fbe] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:14.7: [8086:2fbf] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:15.0: [8086:2fb4] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:15.1: [8086:2fb5] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:15.2: [8086:2fb6] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:15.3: [8086:2fb7] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:16.0: [8086:2f68] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:16.6: [8086:2f6e] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:16.7: [8086:2f6f] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:17.0: [8086:2fd0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:17.4: [8086:2fb8] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:17.5: [8086:2fb9] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:17.6: [8086:2fba] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:17.7: [8086:2fbb] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:1e.0: [8086:2f98] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:1e.1: [8086:2f99] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:1e.2: [8086:2f9a] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:1e.3: [8086:2fc0] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:1e.4: [8086:2f9c] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:1f.0: [8086:2f88] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:7f:1f.2: [8086:2f8a] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:00: PCIe AER handled by firmware Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration Sep 03 14:34:39 oak-md1-s2 kernel: PCI host bridge to bus 0000:00 Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [bus 00-7e] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [io 0x0000-0x03bb window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [io 0x03bc-0x03df window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [io 0x1000-0x7fff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [mem 0x90000000-0xc7ffbfff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: root bus resource [mem 0x38000000000-0x3bfffffffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:00.0: [8086:2f00] type 00 class 0x060000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: [8086:2f02] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: [8086:2f04] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: [8086:2f08] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: [8086:2f09] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.2: [8086:2f0a] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.2: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.2: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:05.0: [8086:2f28] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:05.1: [8086:2f29] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:05.2: [8086:2f2a] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:05.4: [8086:2f2c] type 00 class 0x080020 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:05.4: reg 0x10: [mem 0x91f04000-0x91f04fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.0: reg 0x30: [mem 0xffff0000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: [8086:8d62] type 00 class 0x010601 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: reg 0x10: [io 0x4078-0x407f] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: reg 0x14: [io 0x408c-0x408f] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: reg 0x18: [io 0x4070-0x4077] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: reg 0x1c: [io 0x4088-0x408b] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: reg 0x20: [io 0x4040-0x405f] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: reg 0x24: [mem 0x91f01000-0x91f017ff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.4: PME# supported from D3hot Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:16.0: reg 0x10: [mem 0x3bffff03000-0x3bffff0300f 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:16.1: reg 0x10: [mem 0x3bffff02000-0x3bffff0200f 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1a.0: reg 0x10: [mem 0x91f03000-0x91f033ff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: [8086:8d1e] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1d.0: reg 0x10: [mem 0x91f02000-0x91f023ff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: [8086:8d02] type 00 class 0x010601 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: reg 0x10: [io 0x4068-0x406f] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: reg 0x14: [io 0x4084-0x4087] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: reg 0x18: [io 0x4060-0x4067] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: reg 0x1c: [io 0x4080-0x4083] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: reg 0x20: [io 0x4020-0x403f] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: reg 0x24: [mem 0x91f00000-0x91f007ff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1f.2: PME# supported from D3hot Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: [1000:005f] type 00 class 0x010400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: reg 0x10: [io 0x3000-0x30ff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: reg 0x14: [mem 0x91e00000-0x91e0ffff 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: reg 0x1c: [mem 0x91d00000-0x91dfffff 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: supports D1 D2 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: PCI bridge to [bus 03] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: bridge window [io 0x3000-0x3fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: bridge window [mem 0x91d00000-0x91efffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: [1000:0097] type 00 class 0x010700 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: reg 0x10: [io 0x2000-0x20ff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: reg 0x14: [mem 0x91c40000-0x91c4ffff 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: reg 0x1c: [mem 0x91c00000-0x91c3ffff 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: supports D1 D2 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: reg 0x174: [mem 0x00000000-0x0000ffff 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR0 for 16 VFs) Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: reg 0x17c: [mem 0x00000000-0x0003ffff 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: VF(n) BAR2 space: [mem 0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs) Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: PCI bridge to [bus 04] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: bridge window [io 0x2000-0x2fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: bridge window [mem 0x91c00000-0x91cfffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: [14e4:165f] type 00 class 0x020000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: reg 0x10: [mem 0x91a30000-0x91a3ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: reg 0x18: [mem 0x91a40000-0x91a4ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: reg 0x20: [mem 0x91a50000-0x91a5ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: [14e4:165f] type 00 class 0x020000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: reg 0x10: [mem 0x91a00000-0x91a0ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: reg 0x18: [mem 0x91a10000-0x91a1ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: reg 0x20: [mem 0x91a20000-0x91a2ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: PCI bridge to [bus 02] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: [14e4:165f] type 00 class 0x020000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: reg 0x10: [mem 0x91b30000-0x91b3ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: reg 0x18: [mem 0x91b40000-0x91b4ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: reg 0x20: [mem 0x91b50000-0x91b5ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: [14e4:165f] type 00 class 0x020000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: reg 0x10: [mem 0x91b00000-0x91b0ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: reg 0x18: [mem 0x91b10000-0x91b1ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: reg 0x20: [mem 0x91b20000-0x91b2ffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: bridge window [mem 0x91b00000-0x91bfffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.2: PCI bridge to [bus 05] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.0: PCI bridge to [bus 06] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: [1912:001d] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: PCI bridge to [bus 07-0b] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: bridge window [mem 0x91000000-0x919fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: [1912:001d] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: PCI bridge to [bus 08-0b] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: bridge window [mem 0x91000000-0x919fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: [1912:001a] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: PCI bridge to [bus 09-0a] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:0a:00.0: [102b:0534] type 00 class 0x030000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:0a:00.0: reg 0x10: [mem 0x90000000-0x90ffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:0a:00.0: reg 0x14: [mem 0x91800000-0x91803fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:0a:00.0: reg 0x18: [mem 0x91000000-0x917fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: PCI bridge to [bus 0a] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: bridge window [mem 0x91000000-0x918fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: on NUMA node 0 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 11 12 14 *15) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 *14 15) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 *6 7 9 10 11 12 14 15) Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:01: PCIe AER handled by firmware Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] Sep 03 14:34:39 oak-md1-s2 kernel: acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration Sep 03 14:34:39 oak-md1-s2 kernel: PCI host bridge to bus 0000:80 Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: root bus resource [bus 80-fe] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: root bus resource [io 0x8000-0xffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: root bus resource [mem 0xc8000000-0xfbffbfff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: root bus resource [mem 0x3c000000000-0x3ffffffffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:01.0: [8086:2f02] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:01.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:01.0: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: [8086:2f08] type 01 class 0x060400 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: PME# supported from D0 D3hot D3cold Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: System wakeup disabled by ACPI Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:05.0: [8086:2f28] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:05.1: [8086:2f29] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:05.2: [8086:2f2a] type 00 class 0x088000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:05.4: [8086:2f2c] type 00 class 0x080020 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:05.4: reg 0x10: [mem 0xc8900000-0xc8900fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:01.0: PCI bridge to [bus 81] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: [15b3:1003] type 00 class 0x028000 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: reg 0x10: [mem 0xc8800000-0xc88fffff 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: reg 0x18: [mem 0xc8000000-0xc87fffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: PCI bridge to [bus 82] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: bridge window [mem 0xc8800000-0xc88fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc87fffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: on NUMA node 1 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Enabled 2 GPEs in block 00 to 3F Sep 03 14:34:39 oak-md1-s2 kernel: vgaarb: device added: PCI:0000:0a:00.0,decodes=io+mem,owns=io+mem,locks=none Sep 03 14:34:39 oak-md1-s2 kernel: vgaarb: loaded Sep 03 14:34:39 oak-md1-s2 kernel: vgaarb: bridge control possible 0000:0a:00.0 Sep 03 14:34:39 oak-md1-s2 kernel: SCSI subsystem initialized Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: bus type USB registered Sep 03 14:34:39 oak-md1-s2 kernel: usbcore: registered new interface driver usbfs Sep 03 14:34:39 oak-md1-s2 kernel: usbcore: registered new interface driver hub Sep 03 14:34:39 oak-md1-s2 kernel: usbcore: registered new device driver usb Sep 03 14:34:39 oak-md1-s2 kernel: PCI: Using ACPI for IRQ routing Sep 03 14:34:39 oak-md1-s2 kernel: PCI: pci_cache_line_size set to 64 bytes Sep 03 14:34:39 oak-md1-s2 kernel: e820: reserve RAM buffer [mem 0x0009c000-0x0009ffff] Sep 03 14:34:39 oak-md1-s2 kernel: e820: reserve RAM buffer [mem 0x67b5c000-0x67ffffff] Sep 03 14:34:39 oak-md1-s2 kernel: e820: reserve RAM buffer [mem 0x7a289000-0x7bffffff] Sep 03 14:34:39 oak-md1-s2 kernel: e820: reserve RAM buffer [mem 0x7bae9000-0x7bffffff] Sep 03 14:34:39 oak-md1-s2 kernel: e820: reserve RAM buffer [mem 0x7bb00000-0x7bffffff] Sep 03 14:34:39 oak-md1-s2 kernel: NetLabel: Initializing Sep 03 14:34:39 oak-md1-s2 kernel: NetLabel: domain hash size = 128 Sep 03 14:34:39 oak-md1-s2 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Sep 03 14:34:39 oak-md1-s2 kernel: NetLabel: unlabeled traffic allowed by default Sep 03 14:34:39 oak-md1-s2 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 Sep 03 14:34:39 oak-md1-s2 kernel: hpet0: 8 comparators, 64-bit 14.318180 MHz counter Sep 03 14:34:39 oak-md1-s2 kernel: amd_nb: Cannot enumerate AMD northbridges Sep 03 14:34:39 oak-md1-s2 kernel: Switched to clocksource hpet Sep 03 14:34:39 oak-md1-s2 kernel: pnp: PnP ACPI init Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: bus type PNP registered Sep 03 14:34:39 oak-md1-s2 kernel: pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [io 0x0500-0x053f] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [io 0x0400-0x047f] could not be reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [io 0x0540-0x057f] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [io 0x0600-0x061f] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [io 0x0ca0-0x0ca5] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [io 0x0880-0x0883] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [io 0x0800-0x081f] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [mem 0xfeda8000-0xfedcbfff] could not be reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [mem 0xff000000-0xffffffff] could not be reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) Sep 03 14:34:39 oak-md1-s2 kernel: pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) Sep 03 14:34:39 oak-md1-s2 kernel: pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) Sep 03 14:34:39 oak-md1-s2 kernel: system 00:04: [io 0x0ca8] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:04: [io 0x0cac] has been reserved Sep 03 14:34:39 oak-md1-s2 kernel: system 00:04: Plug and Play ACPI device, IDs IPI0001 PNP0c01 (active) Sep 03 14:34:39 oak-md1-s2 kernel: pnp: PnP ACPI: found 5 devices Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: bus type PNP unregistered Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.0: can't claim BAR 6 [mem 0xffff0000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: BAR 14: assigned [mem 0x92000000-0x920fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: BAR 14: assigned [mem 0x92100000-0x921fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:11.0: BAR 6: assigned [mem 0x91f10000-0x91f1ffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: BAR 6: no space for [mem size 0x00100000 pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: PCI bridge to [bus 03] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: bridge window [io 0x3000-0x3fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:01.0: bridge window [mem 0x91d00000-0x91efffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: BAR 6: no space for [mem size 0x00100000 pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: BAR 9: no space for [mem size 0x00400000 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: BAR 9: failed to assign [mem size 0x00400000 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: BAR 7: no space for [mem size 0x00100000 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: PCI bridge to [bus 04] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: bridge window [io 0x2000-0x2fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:02.0: bridge window [mem 0x91c00000-0x91cfffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: BAR 6: assigned [mem 0x92000000-0x9203ffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: BAR 6: assigned [mem 0x92040000-0x9207ffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: PCI bridge to [bus 02] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: bridge window [mem 0x92000000-0x920fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: BAR 6: assigned [mem 0x92100000-0x9213ffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: BAR 6: assigned [mem 0x92140000-0x9217ffff pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: PCI bridge to [bus 01] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: bridge window [mem 0x92100000-0x921fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.1: bridge window [mem 0x91b00000-0x91bfffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:03.2: PCI bridge to [bus 05] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.0: PCI bridge to [bus 06] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: PCI bridge to [bus 0a] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: bridge window [mem 0x91000000-0x918fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: PCI bridge to [bus 09-0a] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: PCI bridge to [bus 08-0b] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: bridge window [mem 0x91000000-0x919fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: PCI bridge to [bus 07-0b] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: bridge window [mem 0x91000000-0x919fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: resource 4 [io 0x0000-0x03bb window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: resource 5 [io 0x03bc-0x03df window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: resource 6 [io 0x03e0-0x0cf7 window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: resource 7 [io 0x1000-0x7fff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: resource 9 [mem 0x90000000-0xc7ffbfff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:00: resource 10 [mem 0x38000000000-0x3bfffffffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:03: resource 0 [io 0x3000-0x3fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:03: resource 1 [mem 0x91d00000-0x91efffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:04: resource 0 [io 0x2000-0x2fff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:04: resource 1 [mem 0x91c00000-0x91cfffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:02: resource 1 [mem 0x92000000-0x920fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:02: resource 2 [mem 0x91a00000-0x91afffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:01: resource 1 [mem 0x92100000-0x921fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:01: resource 2 [mem 0x91b00000-0x91bfffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:07: resource 1 [mem 0x91000000-0x919fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:07: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:08: resource 1 [mem 0x91000000-0x919fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:08: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:09: resource 1 [mem 0x91000000-0x918fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:09: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:0a: resource 1 [mem 0x91000000-0x918fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:0a: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:01.0: PCI bridge to [bus 81] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: BAR 6: no space for [mem size 0x00100000 pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: PCI bridge to [bus 82] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: bridge window [mem 0xc8800000-0xc88fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc87fffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: resource 4 [io 0x8000-0xffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: resource 5 [mem 0xc8000000-0xfbffbfff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:80: resource 6 [mem 0x3c000000000-0x3ffffffffff window] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:82: resource 1 [mem 0xc8800000-0xc88fffff] Sep 03 14:34:39 oak-md1-s2 kernel: pci_bus 0000:82: resource 2 [mem 0xc8000000-0xc87fffff 64bit pref] Sep 03 14:34:39 oak-md1-s2 kernel: NET: Registered protocol family 2 Sep 03 14:34:39 oak-md1-s2 kernel: TCP established hash table entries: 524288 (order: 10, 4194304 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: TCP: Hash tables configured (established 524288 bind 65536) Sep 03 14:34:39 oak-md1-s2 kernel: TCP: reno registered Sep 03 14:34:39 oak-md1-s2 kernel: UDP hash table entries: 65536 (order: 9, 2097152 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: NET: Registered protocol family 1 Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:0a:00.0: Boot video device Sep 03 14:34:39 oak-md1-s2 kernel: PCI: CLS 32 bytes, default 64 Sep 03 14:34:39 oak-md1-s2 kernel: Unpacking initramfs... Sep 03 14:34:39 oak-md1-s2 kernel: Freeing initrd memory: 23820k freed Sep 03 14:34:39 oak-md1-s2 kernel: DMAR: [Firmware Bug]: RMRR entry for device 03:00.0 is broken - applying workaround Sep 03 14:34:39 oak-md1-s2 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) Sep 03 14:34:39 oak-md1-s2 kernel: software IO TLB [mem 0x76289000-0x7a289000] (64MB) mapped at [ffff880076289000-ffff88007a288fff] Sep 03 14:34:39 oak-md1-s2 kernel: Intel CQM monitoring enabled Sep 03 14:34:39 oak-md1-s2 kernel: RAPL PMU: API unit is 2^-32 Joules, 4 fixed counters, 655360 ms ovfl timer Sep 03 14:34:39 oak-md1-s2 kernel: RAPL PMU: hw unit of domain pp0-core 2^-14 Joules Sep 03 14:34:39 oak-md1-s2 kernel: RAPL PMU: hw unit of domain package 2^-14 Joules Sep 03 14:34:39 oak-md1-s2 kernel: RAPL PMU: hw unit of domain dram 2^-14 Joules Sep 03 14:34:39 oak-md1-s2 kernel: RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules Sep 03 14:34:39 oak-md1-s2 kernel: sha1_ssse3: Using AVX optimized SHA-1 implementation Sep 03 14:34:39 oak-md1-s2 kernel: sha256_ssse3: Using AVX2 optimized SHA-256 implementation Sep 03 14:34:39 oak-md1-s2 kernel: futex hash table entries: 131072 (order: 11, 8388608 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: Initialise system trusted keyring Sep 03 14:34:39 oak-md1-s2 kernel: audit: initializing netlink socket (disabled) Sep 03 14:34:39 oak-md1-s2 kernel: type=2000 audit(1536010469.503:1): initialized Sep 03 14:34:39 oak-md1-s2 kernel: HugeTLB registered 1 GB page size, pre-allocated 0 pages Sep 03 14:34:39 oak-md1-s2 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Sep 03 14:34:39 oak-md1-s2 kernel: zpool: loaded Sep 03 14:34:39 oak-md1-s2 kernel: zbud: loaded Sep 03 14:34:39 oak-md1-s2 kernel: VFS: Disk quotas dquot_6.6.0 Sep 03 14:34:39 oak-md1-s2 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Sep 03 14:34:39 oak-md1-s2 kernel: msgmni has been set to 32768 Sep 03 14:34:39 oak-md1-s2 kernel: Key type big_key registered Sep 03 14:34:39 oak-md1-s2 kernel: SELinux: Registering netfilter hooks Sep 03 14:34:39 oak-md1-s2 kernel: NET: Registered protocol family 38 Sep 03 14:34:39 oak-md1-s2 kernel: Key type asymmetric registered Sep 03 14:34:39 oak-md1-s2 kernel: Asymmetric key parser 'x509' registered Sep 03 14:34:39 oak-md1-s2 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) Sep 03 14:34:39 oak-md1-s2 kernel: io scheduler noop registered Sep 03 14:34:39 oak-md1-s2 kernel: io scheduler deadline registered (default) Sep 03 14:34:39 oak-md1-s2 kernel: io scheduler cfq registered Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:01.0: irq 25 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:02.0: irq 26 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:03.0: irq 27 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:03.1: irq 28 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:03.2: irq 29 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:1c.0: irq 30 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:1c.7: irq 31 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:80:01.0: irq 33 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:80:03.0: irq 34 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:03:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:04:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:00:02.0:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:02:00.1: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:01:00.1: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:00:03.1:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:03.2: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:00:03.2:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:07:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:08:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:09:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:0a:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:00:1c.7:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:80:01.0:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pci 0000:82:00.0: Signaling PME through PCIe PME interrupt Sep 03 14:34:39 oak-md1-s2 kernel: pcie_pme 0000:80:03.0:pcie01: service driver pcie_pme loaded Sep 03 14:34:39 oak-md1-s2 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Sep 03 14:34:39 oak-md1-s2 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Sep 03 14:34:39 oak-md1-s2 kernel: intel_idle: MWAIT substates: 0x2120 Sep 03 14:34:39 oak-md1-s2 kernel: intel_idle: v0.4.1 model 0x3F Sep 03 14:34:39 oak-md1-s2 kernel: intel_idle: lapic_timer_reliable_states 0xffffffff Sep 03 14:34:39 oak-md1-s2 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 Sep 03 14:34:39 oak-md1-s2 kernel: ACPI: Power Button [PWRF] Sep 03 14:34:39 oak-md1-s2 kernel: ERST: Error Record Serialization Table (ERST) support is initialized. Sep 03 14:34:39 oak-md1-s2 kernel: pstore: Registered erst as persistent store backend Sep 03 14:34:39 oak-md1-s2 kernel: GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. Sep 03 14:34:39 oak-md1-s2 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Sep 03 14:34:39 oak-md1-s2 kernel: 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A Sep 03 14:34:39 oak-md1-s2 kernel: 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Sep 03 14:34:39 oak-md1-s2 kernel: Non-volatile memory driver v1.3 Sep 03 14:34:39 oak-md1-s2 kernel: Linux agpgart interface v0.103 Sep 03 14:34:39 oak-md1-s2 kernel: crash memory driver: version 1.1 Sep 03 14:34:39 oak-md1-s2 kernel: rdac: device handler registered Sep 03 14:34:39 oak-md1-s2 kernel: hp_sw: device handler registered Sep 03 14:34:39 oak-md1-s2 kernel: emc: device handler registered Sep 03 14:34:39 oak-md1-s2 kernel: alua: device handler registered Sep 03 14:34:39 oak-md1-s2 kernel: libphy: Fixed MDIO Bus: probed Sep 03 14:34:39 oak-md1-s2 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci: EHCI PCI platform driver Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1a.0: EHCI Host Controller Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1a.0: debug port 2 Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1a.0: cache line size of 32 is not supported Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1a.0: irq 18, io mem 0x91f03000 Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 Sep 03 14:34:39 oak-md1-s2 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 Sep 03 14:34:39 oak-md1-s2 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Sep 03 14:34:39 oak-md1-s2 kernel: usb usb1: Product: EHCI Host Controller Sep 03 14:34:39 oak-md1-s2 kernel: usb usb1: Manufacturer: Linux 3.10.0-693.2.2.el7_lustre.pl1.x86_64 ehci_hcd Sep 03 14:34:39 oak-md1-s2 kernel: usb usb1: SerialNumber: 0000:00:1a.0 Sep 03 14:34:39 oak-md1-s2 kernel: hub 1-0:1.0: USB hub found Sep 03 14:34:39 oak-md1-s2 kernel: hub 1-0:1.0: 2 ports detected Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1d.0: EHCI Host Controller Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1d.0: debug port 2 Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1d.0: cache line size of 32 is not supported Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1d.0: irq 18, io mem 0x91f02000 Sep 03 14:34:39 oak-md1-s2 kernel: ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 Sep 03 14:34:39 oak-md1-s2 kernel: usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 Sep 03 14:34:39 oak-md1-s2 kernel: usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Sep 03 14:34:39 oak-md1-s2 kernel: usb usb2: Product: EHCI Host Controller Sep 03 14:34:39 oak-md1-s2 kernel: usb usb2: Manufacturer: Linux 3.10.0-693.2.2.el7_lustre.pl1.x86_64 ehci_hcd Sep 03 14:34:39 oak-md1-s2 kernel: usb usb2: SerialNumber: 0000:00:1d.0 Sep 03 14:34:39 oak-md1-s2 kernel: hub 2-0:1.0: USB hub found Sep 03 14:34:39 oak-md1-s2 kernel: hub 2-0:1.0: 2 ports detected Sep 03 14:34:39 oak-md1-s2 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Sep 03 14:34:39 oak-md1-s2 kernel: ohci-pci: OHCI PCI platform driver Sep 03 14:34:39 oak-md1-s2 kernel: uhci_hcd: USB Universal Host Controller Interface driver Sep 03 14:34:39 oak-md1-s2 kernel: usbcore: registered new interface driver usbserial Sep 03 14:34:39 oak-md1-s2 kernel: usbcore: registered new interface driver usbserial_generic Sep 03 14:34:39 oak-md1-s2 kernel: usbserial: USB Serial support registered for generic Sep 03 14:34:39 oak-md1-s2 kernel: i8042: PNP: No PS/2 controller found. Probing ports directly. Sep 03 14:34:39 oak-md1-s2 kernel: tsc: Refined TSC clocksource calibration: 3399.997 MHz Sep 03 14:34:39 oak-md1-s2 kernel: i8042: No controller found Sep 03 14:34:39 oak-md1-s2 kernel: Switched to clocksource tsc Sep 03 14:34:39 oak-md1-s2 kernel: mousedev: PS/2 mouse device common for all mice Sep 03 14:34:39 oak-md1-s2 kernel: rtc_cmos 00:00: RTC can wake from S4 Sep 03 14:34:39 oak-md1-s2 kernel: rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 Sep 03 14:34:39 oak-md1-s2 kernel: rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs Sep 03 14:34:39 oak-md1-s2 kernel: Intel P-state driver initializing. Sep 03 14:34:39 oak-md1-s2 kernel: cpuidle: using governor menu Sep 03 14:34:39 oak-md1-s2 kernel: hidraw: raw HID events driver (C) Jiri Kosina Sep 03 14:34:39 oak-md1-s2 kernel: usbcore: registered new interface driver usbhid Sep 03 14:34:39 oak-md1-s2 kernel: usbhid: USB HID core driver Sep 03 14:34:39 oak-md1-s2 kernel: drop_monitor: Initializing network drop monitor service Sep 03 14:34:39 oak-md1-s2 kernel: TCP: cubic registered Sep 03 14:34:39 oak-md1-s2 kernel: Initializing XFRM netlink socket Sep 03 14:34:39 oak-md1-s2 kernel: NET: Registered protocol family 10 Sep 03 14:34:39 oak-md1-s2 kernel: NET: Registered protocol family 17 Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU0 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU1 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU2 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU3 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU4 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU5 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU6 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU7 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU8 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU9 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU10 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU11 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU12 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU13 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU14 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU15 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU16 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU17 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU18 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU19 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU20 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU21 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU22 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: CPU23 sig=0x306f2, pf=0x1, revision=0x3d Sep 03 14:34:39 oak-md1-s2 kernel: microcode: Microcode Update Driver: v2.01 , Peter Oruba Sep 03 14:34:39 oak-md1-s2 kernel: PM: Hibernation image not present or could not be loaded. Sep 03 14:34:39 oak-md1-s2 kernel: Loading compiled-in X.509 certificates Sep 03 14:34:39 oak-md1-s2 kernel: Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' Sep 03 14:34:39 oak-md1-s2 kernel: Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' Sep 03 14:34:39 oak-md1-s2 kernel: Loaded X.509 cert 'CentOS Linux kernel signing key: ac6b81e043e4e272f8cb40e32ac29ef2559d586f' Sep 03 14:34:39 oak-md1-s2 kernel: registered taskstats version 1 Sep 03 14:34:39 oak-md1-s2 kernel: Key type trusted registered Sep 03 14:34:39 oak-md1-s2 kernel: Key type encrypted registered Sep 03 14:34:39 oak-md1-s2 kernel: IMA: No TPM chip found, activating TPM-bypass! Sep 03 14:34:39 oak-md1-s2 kernel: Magic number: 2:501:600 Sep 03 14:34:39 oak-md1-s2 kernel: memory memory589: hash matches Sep 03 14:34:39 oak-md1-s2 kernel: rtc_cmos 00:00: setting system clock to 2018-09-03 21:34:38 UTC (1536010478) Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1: new high-speed USB device number 2 using ehci-pci Sep 03 14:34:39 oak-md1-s2 kernel: usb 2-1: new high-speed USB device number 2 using ehci-pci Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1: New USB device found, idVendor=8087, idProduct=800a Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Sep 03 14:34:39 oak-md1-s2 kernel: hub 1-1:1.0: USB hub found Sep 03 14:34:39 oak-md1-s2 kernel: hub 1-1:1.0: 6 ports detected Sep 03 14:34:39 oak-md1-s2 kernel: usb 2-1: New USB device found, idVendor=8087, idProduct=8002 Sep 03 14:34:39 oak-md1-s2 kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Sep 03 14:34:39 oak-md1-s2 kernel: hub 2-1:1.0: USB hub found Sep 03 14:34:39 oak-md1-s2 kernel: hub 2-1:1.0: 8 ports detected Sep 03 14:34:39 oak-md1-s2 kernel: Freeing unused kernel memory: 1764k freed Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1.6: new high-speed USB device number 3 using ehci-pci Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: random: systemd: uninitialized urandom read (16 bytes read) Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1.6: New USB device found, idVendor=413c, idProduct=a001 Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3 Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1.6: Product: Gadget USB HUB Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1.6: Manufacturer: no manufacturer Sep 03 14:34:39 oak-md1-s2 kernel: usb 1-1.6: SerialNumber: 0123456789 Sep 03 14:34:39 oak-md1-s2 kernel: hub 1-1.6:1.0: USB hub found Sep 03 14:34:39 oak-md1-s2 kernel: hub 1-1.6:1.0: 6 ports detected Sep 03 14:34:39 oak-md1-s2 systemd[1]: Starting Create list of required static device nodes for the current kernel... Sep 03 14:34:39 oak-md1-s2 systemd[1]: Started Create list of required static device nodes for the current kernel. Sep 03 14:34:39 oak-md1-s2 kernel: pps_core: LinuxPPS API ver. 1 registered Sep 03 14:34:39 oak-md1-s2 kernel: pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti Sep 03 14:34:39 oak-md1-s2 kernel: megasas: 07.701.17.00-rh1 Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: FW now in Ready state Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 36 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 37 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 38 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 39 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 40 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 41 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 42 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: PTP clock support registered Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 43 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 44 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 45 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 46 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 47 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 48 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 49 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 50 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 51 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 52 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 53 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 54 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 55 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 56 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 57 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 58 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: irq 59 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: firmware supports msix : (96) Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: current msix/online cpus : (24/24) Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: RDPQ mode : (disabled) Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: Current firmware maximum commands: 928 LDIO threshold: 237 Sep 03 14:34:39 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: FW supports sync cache : No Sep 03 14:34:39 oak-md1-s2 kernel: libata version 3.00 loaded. Sep 03 14:34:39 oak-md1-s2 kernel: mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014) Sep 03 14:34:39 oak-md1-s2 kernel: mlx4_core: Initializing 0000:82:00.0 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas version 15.100.00.00 loaded Sep 03 14:34:39 oak-md1-s2 kernel: [drm] Initialized Sep 03 14:34:39 oak-md1-s2 kernel: random: fast init done Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (131589404 kB) Sep 03 14:34:39 oak-md1-s2 kernel: tg3.c:v3.137 (May 11, 2014) Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:11.4: version 3.0 Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:11.4: irq 63 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:11.4: SSS flag set, parallel bus scan disabled Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f2 Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:11.4: flags: 64bit ncq stag led clo pio slum part ems apst Sep 03 14:34:39 oak-md1-s2 kernel: scsi host2: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host3: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host4: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host5: ahci Sep 03 14:34:39 oak-md1-s2 kernel: ata1: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01100 irq 63 Sep 03 14:34:39 oak-md1-s2 kernel: ata2: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01180 irq 63 Sep 03 14:34:39 oak-md1-s2 kernel: ata3: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01200 irq 63 Sep 03 14:34:39 oak-md1-s2 kernel: ata4: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01280 irq 63 Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:1f.2: irq 65 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:1f.2: SSS flag set, parallel bus scan disabled Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f3 Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:01:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode Sep 03 14:34:39 oak-md1-s2 kernel: ahci 0000:00:1f.2: flags: 64bit ncq stag led clo pio slum part ems apst Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.0 eth2: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f4 Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.0 eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.0 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.0 eth2: dma_rwctrl[00000001] dma_mask[64-bit] Sep 03 14:34:39 oak-md1-s2 kernel: scsi host6: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host7: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host8: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host9: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host10: ahci Sep 03 14:34:39 oak-md1-s2 kernel: scsi host11: ahci Sep 03 14:34:39 oak-md1-s2 kernel: ata5: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00100 irq 65 Sep 03 14:34:39 oak-md1-s2 kernel: ata6: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00180 irq 65 Sep 03 14:34:39 oak-md1-s2 kernel: ata7: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00200 irq 65 Sep 03 14:34:39 oak-md1-s2 kernel: ata8: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00280 irq 65 Sep 03 14:34:39 oak-md1-s2 kernel: ata9: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00300 irq 65 Sep 03 14:34:39 oak-md1-s2 kernel: ata10: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00380 irq 65 Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.1 eth3: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f5 Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.1 eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.1 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] Sep 03 14:34:39 oak-md1-s2 kernel: tg3 0000:02:00.1 eth3: dma_rwctrl[00000001] dma_mask[64-bit] Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas_cm0: MSI-X vectors supported: 96, no of cores: 24, max_msix_vectors: -1 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 68 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 69 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 70 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 71 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 74 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 75 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 76 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 77 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 78 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 79 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 80 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 81 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 82 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 83 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 84 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 85 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 86 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 87 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 88 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 89 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 90 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 91 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 92 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas 0000:04:00.0: irq 93 for MSI/MSI-X Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 68 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 69 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 70 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 71 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 74 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 75 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 76 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 77 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 78 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 79 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 80 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 81 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 82 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 83 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 84 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 85 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 86 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 87 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 88 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 89 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 90 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 91 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 92 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 93 Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas_cm0: iomem(0x0000000091c40000), mapped(0xffffc9000d780000), size(65536) Sep 03 14:34:39 oak-md1-s2 kernel: mpt3sas_cm0: ioport(0x0000000000002000), size(256) Sep 03 14:34:40 oak-md1-s2 kernel: mpt3sas_cm0: Allocated physical memory: size(19554 kB) Sep 03 14:34:40 oak-md1-s2 kernel: mpt3sas_cm0: Current Controller Queue Depth(9564),Max Controller Queue Depth(9664) Sep 03 14:34:40 oak-md1-s2 kernel: mpt3sas_cm0: Scatter Gather Elements per IO(128) Sep 03 14:34:40 oak-md1-s2 kernel: mpt3sas_cm0: LSISAS3008: FWVersion(16.00.01.00), ChipRevision(0x02), BiosVersion(04.00.00.00) Sep 03 14:34:40 oak-md1-s2 kernel: mpt3sas_cm0: Protocol=( Sep 03 14:34:40 oak-md1-s2 kernel: Initiator Sep 03 14:34:40 oak-md1-s2 kernel: ,Target Sep 03 14:34:40 oak-md1-s2 kernel: ), Sep 03 14:34:40 oak-md1-s2 kernel: Capabilities=( Sep 03 14:34:40 oak-md1-s2 kernel: TLR Sep 03 14:34:40 oak-md1-s2 kernel: ,EEDP Sep 03 14:34:40 oak-md1-s2 kernel: ,Snapshot Buffer Sep 03 14:34:40 oak-md1-s2 kernel: ,Diag Trace Buffer Sep 03 14:34:40 oak-md1-s2 kernel: ,Task Set Full Sep 03 14:34:40 oak-md1-s2 kernel: ,NCQ Sep 03 14:34:40 oak-md1-s2 kernel: ) Sep 03 14:34:40 oak-md1-s2 kernel: scsi host1: Fusion MPT SAS Host Sep 03 14:34:40 oak-md1-s2 kernel: mpt3sas_cm0: sending port enable !! Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: Init cmd success Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: firmware type : Legacy(64 VD) firmware Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: controller type : iMR(0MB) Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: Online Controller Reset(OCR) : Enabled Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: Secure JBOD support : No Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: INIT adapter done Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: Jbod map is not supported megasas_setup_jbod_map 5121 Sep 03 14:34:40 oak-md1-s2 kernel: ata1: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: unevenspan support : yes Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: firmware crash dump : no Sep 03 14:34:40 oak-md1-s2 kernel: megaraid_sas 0000:03:00.0: jbod sync map : no Sep 03 14:34:40 oak-md1-s2 kernel: scsi host0: Avago SAS based MegaRAID driver Sep 03 14:34:40 oak-md1-s2 kernel: ata5: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:40 oak-md1-s2 kernel: scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.28 PQ: 0 ANSI: 5 Sep 03 14:34:40 oak-md1-s2 kernel: ata2: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:40 oak-md1-s2 kernel: ata3: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:40 oak-md1-s2 kernel: ata4: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:41 oak-md1-s2 kernel: ata6: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:41 oak-md1-s2 kernel: ata7: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:41 oak-md1-s2 kernel: mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aafc090), phys(8) Sep 03 14:34:41 oak-md1-s2 kernel: scsi 1:0:0:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Sep 03 14:34:41 oak-md1-s2 kernel: scsi 1:0:0:0: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) Sep 03 14:34:41 oak-md1-s2 kernel: scsi 1:0:0:0: SSP: enclosure_logical_id(0x500605b00aafc090), slot(3) Sep 03 14:34:41 oak-md1-s2 kernel: scsi 1:0:0:0: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:41 oak-md1-s2 kernel: scsi 1:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:41 oak-md1-s2 kernel: ata8: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:42 oak-md1-s2 kernel: ata9: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:42 oak-md1-s2 kernel: ata10: SATA link down (SStatus 0 SControl 300) Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: PCIe link width is x8, device supports x8 Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 94 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 95 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 96 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 97 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 98 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 99 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 100 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 101 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 102 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 103 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 104 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 105 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 106 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 107 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 108 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 109 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 110 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 111 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 112 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 113 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 114 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 115 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 116 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 117 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_core 0000:82:00.0: irq 118 for MSI/MSI-X Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:1: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:1: SSP: enclosure_logical_id(0x500605b00aafc090), slot(3) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:1: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:46 oak-md1-s2 kernel: mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:2: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:2: SSP: enclosure_logical_id(0x500605b00aafc090), slot(3) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:2: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:31: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:31: SSP: enclosure_logical_id(0x500605b00aafc090), slot(3) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:31: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:0: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:0: SSP: enclosure_logical_id(0x500605b00aafc090), slot(7) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:0: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:1: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:1: SSP: enclosure_logical_id(0x500605b00aafc090), slot(7) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:1: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:2: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:2: SSP: enclosure_logical_id(0x500605b00aafc090), slot(7) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:2: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:31: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:31: SSP: enclosure_logical_id(0x500605b00aafc090), slot(7) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:31: SSP: enclosure level(0x0000), connector name( ) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) Sep 03 14:34:46 oak-md1-s2 kernel: mpt3sas_cm0: port enable: SUCCESS Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:0: rdac: LUN 0 (IOSHIP) (owned) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:1: rdac: LUN 1 (IOSHIP) (unowned) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:0:2: rdac: LUN 2 (IOSHIP) (owned) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:0: rdac: LUN 0 (IOSHIP) (unowned) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:1: rdac: LUN 1 (IOSHIP) (owned) Sep 03 14:34:46 oak-md1-s2 kernel: scsi 1:0:1:2: rdac: LUN 2 (IOSHIP) (unowned) Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:1:0: [sdd] 574402560 512-byte logical blocks: (294 GB/273 GiB) Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:0: [sda] 574402560 512-byte logical blocks: (294 GB/273 GiB) Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:1: [sdb] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) Sep 03 14:34:46 oak-md1-s2 kernel: sd 0:2:0:0: [sdg] 487325696 512-byte logical blocks: (249 GB/232 GiB) Sep 03 14:34:46 oak-md1-s2 kernel: sd 0:2:0:0: [sdg] Write Protect is off Sep 03 14:34:46 oak-md1-s2 kernel: sd 0:2:0:0: [sdg] Mode Sense: 1f 00 10 08 Sep 03 14:34:46 oak-md1-s2 kernel: sd 0:2:0:0: [sdg] Write cache: disabled, read cache: disabled, supports DPO and FUA Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:0: [sda] Write Protect is off Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:0: [sda] Mode Sense: 83 00 10 08 Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:1: [sdb] Write Protect is off Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:1: [sdb] Mode Sense: 83 00 10 08 Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA Sep 03 14:34:46 oak-md1-s2 kernel: sd 1:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:2: [sdf] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:0:2: [sdc] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:1: [sde] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:0:2: [sdc] Write Protect is off Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:0:2: [sdc] Mode Sense: 83 00 10 08 Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:1: [sde] Write Protect is off Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:1: [sde] Mode Sense: 83 00 10 08 Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:0:2: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:2: [sdf] Write Protect is off Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:2: [sdf] Mode Sense: 83 00 10 08 Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:1: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:2: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:0:0: [sda] Attached SCSI disk Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:0:1: [sdb] Attached SCSI disk Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:0:2: [sdc] Attached SCSI disk Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:2: [sdf] Attached SCSI disk Sep 03 14:34:47 oak-md1-s2 kernel: sdg: sdg1 sdg2 Sep 03 14:34:47 oak-md1-s2 kernel: sd 0:2:0:0: [sdg] Attached SCSI disk Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:1: [sde] Attached SCSI disk Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:0: [sdd] Write Protect is off Sep 03 14:34:47 oak-md1-s2 kernel: random: crng init done Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:0: [sdd] Mode Sense: 83 00 10 08 Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA Sep 03 14:34:47 oak-md1-s2 kernel: sd 1:0:1:0: [sdd] Attached SCSI disk Sep 03 14:34:47 oak-md1-s2 kernel: EXT4-fs (sdg1): mounted filesystem with ordered data mode. Opts: (null) Sep 03 14:34:47 oak-md1-s2 systemd[1]: Stopped Create list of required static device nodes for the current kernel. Sep 03 14:34:47 oak-md1-s2 systemd[1]: Stopping Create list of required static device nodes for the current kernel... Sep 03 14:34:49 oak-md1-s2 kernel: SELinux: Disabled at runtime. Sep 03 14:34:49 oak-md1-s2 kernel: SELinux: Unregistering netfilter hooks Sep 03 14:34:49 oak-md1-s2 kernel: type=1404 audit(1536010488.062:2): selinux=0 auid=4294967295 ses=4294967295 Sep 03 14:34:49 oak-md1-s2 kernel: ip_tables: (C) 2000-2006 Netfilter Core Team Sep 03 14:34:49 oak-md1-s2 kernel: EXT4-fs (sdg1): re-mounted. Opts: (null) Sep 03 14:34:49 oak-md1-s2 kernel: device-mapper: uevent: version 1.0.3 Sep 03 14:34:49 oak-md1-s2 kernel: device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com Sep 03 14:34:49 oak-md1-s2 kernel: type=1305 audit(1536010489.750:3): audit_pid=650 old=0 auid=4294967295 ses=4294967295 res=1 Sep 03 14:34:49 oak-md1-s2 kernel: RPC: Registered named UNIX socket transport module. Sep 03 14:34:49 oak-md1-s2 kernel: RPC: Registered udp transport module. Sep 03 14:34:49 oak-md1-s2 kernel: RPC: Registered tcp transport module. Sep 03 14:34:49 oak-md1-s2 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Sep 03 14:34:50 oak-md1-s2 systemd[1]: Starting Initialize the iWARP/InfiniBand/RDMA stack in the kernel... Sep 03 14:34:50 oak-md1-s2 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Sep 03 14:34:51 oak-md1-s2 kernel: mei_me 0000:00:16.0: Device doesn't have valid ME Interface Sep 03 14:34:51 oak-md1-s2 kernel: ACPI Error: No handler for Region [SYSI] (ffff880169b047e0) [IPMI] (20130517/evregion-162) Sep 03 14:34:51 oak-md1-s2 kernel: ACPI Error: Region IPMI (ID=7) has no handler (20130517/exfldio-305) Sep 03 14:34:51 oak-md1-s2 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff880169b03578), AE_NOT_EXIST (20130517/psparse-536) Sep 03 14:34:51 oak-md1-s2 kernel: ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff880169b034d8), AE_NOT_EXIST (20130517/psparse-536) Sep 03 14:34:51 oak-md1-s2 kernel: ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) Sep 03 14:34:51 oak-md1-s2 kernel: wmi: Mapper loaded Sep 03 14:34:51 oak-md1-s2 kernel: ipmi message handler version 39.2 Sep 03 14:34:51 oak-md1-s2 kernel: sd 1:0:0:0: Attached scsi generic sg0 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: sd 1:0:0:1: Attached scsi generic sg1 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: sd 1:0:0:2: Attached scsi generic sg2 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: scsi 1:0:0:31: Attached scsi generic sg3 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: sd 1:0:1:0: Attached scsi generic sg4 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: sd 1:0:1:1: Attached scsi generic sg5 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: sd 1:0:1:2: Attached scsi generic sg6 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: ipmi device interface Sep 03 14:34:51 oak-md1-s2 kernel: scsi 1:0:1:31: Attached scsi generic sg7 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: sd 0:2:0:0: Attached scsi generic sg8 type 0 Sep 03 14:34:51 oak-md1-s2 kernel: input: PC Speaker as /devices/platform/pcspkr/input/input1 Sep 03 14:34:51 oak-md1-s2 kernel: IPMI System Interface driver. Sep 03 14:34:51 oak-md1-s2 kernel: ipmi_si: probing via SMBIOS Sep 03 14:34:51 oak-md1-s2 kernel: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 Sep 03 14:34:51 oak-md1-s2 kernel: ipmi_si: Adding SMBIOS-specified kcs state machine Sep 03 14:34:51 oak-md1-s2 kernel: ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 Sep 03 14:34:51 oak-md1-s2 kernel: device-mapper: multipath round-robin: version 1.2.0 loaded Sep 03 14:34:51 oak-md1-s2 kernel: (NULL device *): The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. Sep 03 14:34:51 oak-md1-s2 kernel: dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2) Sep 03 14:34:51 oak-md1-s2 kernel: ipmi_si ipmi_si.0: Using irq 10 Sep 03 14:34:51 oak-md1-s2 kernel: ipmi_si ipmi_si.0: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) Sep 03 14:34:51 oak-md1-s2 kernel: ipmi_si ipmi_si.0: IPMI kcs interface initialized Sep 03 14:34:51 oak-md1-s2 kernel: iTCO_vendor_support: vendor-support=0 Sep 03 14:34:51 oak-md1-s2 kernel: iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 Sep 03 14:34:51 oak-md1-s2 kernel: iTCO_wdt: Found a Wellsburg TCO device (Version=2, TCOBASE=0x0460) Sep 03 14:34:51 oak-md1-s2 kernel: iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) Sep 03 14:34:51 oak-md1-s2 kernel: AES CTR mode by8 optimization enabled Sep 03 14:34:51 oak-md1-s2 kernel: alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) Sep 03 14:34:51 oak-md1-s2 kernel: alg: No test for crc32 (crc32-pclmul) Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: Found RAPL domain package Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: Found RAPL domain dram Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: DRAM domain energy unit 15300pj Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: RAPL package 0 domain package locked by BIOS Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: Found RAPL domain package Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: Found RAPL domain dram Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: DRAM domain energy unit 15300pj Sep 03 14:34:52 oak-md1-s2 kernel: intel_rapl: RAPL package 1 domain package locked by BIOS Sep 03 14:34:52 oak-md1-s2 kernel: EDAC MC: Ver: 3.0.0 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fa0 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fa0 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fa0 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2ffc Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2ffc Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2ffc Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2ffd Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2ffd Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2ffd Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f60 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fa8 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fa8 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fa8 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f71 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f71 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f71 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2faa Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2faa Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2faa Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fab Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fab Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fab Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fac Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fac Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fac Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fad Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fad Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fad Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbd Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbd Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbd Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbf Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbf Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbf Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fb9 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fb9 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fb9 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbb Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbb Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2fbb Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f68 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f68 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f68 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f79 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f6a Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f6b Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f6c Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Seeking for: PCI ID 8086:2f6d Sep 03 14:34:52 oak-md1-s2 kernel: EDAC MC0: Giving out device to 'sbridge_edac.c' 'Haswell Socket#0': DEV 0000:7f:12.0 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC MC1: Giving out device to 'sbridge_edac.c' 'Haswell Socket#1': DEV 0000:ff:12.0 Sep 03 14:34:52 oak-md1-s2 kernel: EDAC sbridge: Ver: 1.1.1 Sep 03 14:34:52 oak-md1-s2 kernel: device-mapper: multipath service-time: version 0.3.0 loaded Sep 03 14:34:52 oak-md1-s2 kernel: device-mapper: table: 253:3: multipath: error getting device Sep 03 14:34:52 oak-md1-s2 kernel: device-mapper: ioctl: error adding target to table Sep 03 14:34:53 oak-md1-s2 kernel: Adding 4194300k swap on /dev/sdg2. Priority:-1 extents:1 across:4194300k FS Sep 03 14:34:53 oak-md1-s2 kernel: mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014) Sep 03 14:34:53 oak-md1-s2 kernel: mlx4_ib_add: counter index 0 for port 1 allocated 0 Sep 03 14:34:53 oak-md1-s2 kernel: Loading iSCSI transport class v2.0-870. Sep 03 14:34:53 oak-md1-s2 kernel: iscsi: registered transport (iser) Sep 03 14:34:53 oak-md1-s2 kernel: RPC: Registered rdma transport module. Sep 03 14:34:53 oak-md1-s2 kernel: RPC: Registered rdma backchannel transport module. Sep 03 14:34:53 oak-md1-s2 systemd[1]: Started Initialize the iWARP/InfiniBand/RDMA stack in the kernel. Sep 03 14:34:53 oak-md1-s2 kernel: tg3 0000:01:00.0: irq 119 for MSI/MSI-X Sep 03 14:34:53 oak-md1-s2 kernel: tg3 0000:01:00.0: irq 120 for MSI/MSI-X Sep 03 14:34:53 oak-md1-s2 kernel: tg3 0000:01:00.0: irq 121 for MSI/MSI-X Sep 03 14:34:53 oak-md1-s2 kernel: tg3 0000:01:00.0: irq 122 for MSI/MSI-X Sep 03 14:34:53 oak-md1-s2 kernel: tg3 0000:01:00.0: irq 123 for MSI/MSI-X Sep 03 14:34:53 oak-md1-s2 kernel: IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready Sep 03 14:34:57 oak-md1-s2 kernel: tg3 0000:01:00.0 em1: Link is up at 1000 Mbps, full duplex Sep 03 14:34:57 oak-md1-s2 kernel: tg3 0000:01:00.0 em1: Flow control is off for TX and off for RX Sep 03 14:34:57 oak-md1-s2 kernel: tg3 0000:01:00.0 em1: EEE is enabled Sep 03 14:34:57 oak-md1-s2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready Sep 03 14:34:58 oak-md1-s2 kernel: IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready Sep 03 14:34:58 oak-md1-s2 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready Sep 03 14:35:02 oak-md1-s2 kernel: FS-Cache: Loaded Sep 03 14:35:02 oak-md1-s2 kernel: FS-Cache: Netfs 'nfs' registered for caching Sep 03 14:35:02 oak-md1-s2 kernel: Key type dns_resolver registered Sep 03 14:35:02 oak-md1-s2 kernel: NFS: Registering the id_resolver key type Sep 03 14:35:02 oak-md1-s2 kernel: Key type id_resolver registered Sep 03 14:35:02 oak-md1-s2 kernel: Key type id_legacy registered Sep 03 14:35:02 oak-md1-s2 systemd[1]: Starting Crash recovery kernel arming... Sep 03 14:35:03 oak-md1-s2 kdumpctl[2731]: kexec: loaded kdump kernel Sep 03 14:35:03 oak-md1-s2 systemd[1]: Started Crash recovery kernel arming. Sep 03 14:35:15 oak-md1-s2 kernel: Fusion MPT base driver 3.04.20 Sep 03 14:35:15 oak-md1-s2 kernel: Copyright (c) 1999-2008 LSI Corporation Sep 03 14:35:15 oak-md1-s2 kernel: Fusion MPT misc device (ioctl) driver 3.04.20 Sep 03 14:35:15 oak-md1-s2 kernel: mptctl: Registered with Fusion MPT base driver Sep 03 14:35:15 oak-md1-s2 kernel: mptctl: /dev/mptctl @ (major,minor=10,220) Sep 03 14:35:15 oak-md1-s2 kernel: mpt2sas version 20.103.00.00 loaded Sep 03 14:35:17 oak-md1-s2 systemd[1]: Startup finished in 4.587s (kernel) + 9.072s (initrd) + 29.415s (userspace) = 43.074s. Sep 03 15:16:03 oak-md1-s2 kernel: libcfs: loading out-of-tree module taints kernel. Sep 03 15:16:03 oak-md1-s2 kernel: libcfs: module verification failed: signature and/or required key missing - tainting kernel Sep 03 15:16:03 oak-md1-s2 kernel: LNet: HW NUMA nodes: 2, HW CPU cores: 24, npartitions: 2 Sep 03 15:16:03 oak-md1-s2 kernel: alg: No test for adler32 (adler32-zlib) Sep 03 15:16:03 oak-md1-s2 kernel: alg: No test for crc32 (crc32-table) Sep 03 15:16:04 oak-md1-s2 kernel: Lustre: Lustre: Build Version: 2.10.4 Sep 03 15:16:04 oak-md1-s2 kernel: LNet: Using FMR for registration Sep 03 15:16:04 oak-md1-s2 kernel: LNet: Added LNI 10.0.2.52@o2ib5 [8/256/0/180] Sep 03 15:16:11 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 4297164 seconds Sep 03 15:16:11 oak-md1-s2 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Sep 03 15:16:13 oak-md1-s2 kernel: LustreError: 11-0: oak-OST0003-osc-MDT0000: operation ost_connect to node 10.0.2.102@o2ib5 failed: rc = -16 Sep 03 15:16:14 oak-md1-s2 kernel: LustreError: 11-0: oak-OST0038-osc-MDT0000: operation ost_connect to node 10.0.2.105@o2ib5 failed: rc = -16 Sep 03 15:16:14 oak-md1-s2 kernel: LustreError: Skipped 55 previous similar messages Sep 03 15:16:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Imperative Recovery not enabled, recovery window 300-900 Sep 03 15:16:15 oak-md1-s2 kernel: Lustre: oak-MDD0000: changelog on Sep 03 15:16:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: in recovery but waiting for the first client to connect Sep 03 15:16:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Sep 03 15:16:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Sep 03 15:16:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Sep 03 15:16:32 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Sep 03 15:16:39 oak-md1-s2 kernel: LustreError: 11-0: oak-OST0003-osc-MDT0000: operation ost_connect to node 10.0.2.102@o2ib5 failed: rc = -16 Sep 03 15:16:39 oak-md1-s2 kernel: LustreError: Skipped 135 previous similar messages Sep 03 15:16:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 0@lo) Sep 03 15:16:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Sep 03 15:16:43 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Sep 03 15:16:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.105@o2ib5 (at 10.0.2.105@o2ib5) Sep 03 15:16:47 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Sep 03 15:16:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.101@o2ib5 (at 10.0.2.101@o2ib5) Sep 03 15:16:57 oak-md1-s2 kernel: Lustre: Skipped 31 previous similar messages Sep 03 15:17:04 oak-md1-s2 kernel: LustreError: 11-0: oak-OST0007-osc-MDT0000: operation ost_connect to node 10.0.2.102@o2ib5 failed: rc = -16 Sep 03 15:17:04 oak-md1-s2 kernel: LustreError: Skipped 95 previous similar messages Sep 03 15:17:29 oak-md1-s2 kernel: LustreError: 11-0: oak-OST000f-osc-MDT0000: operation ost_connect to node 10.0.2.102@o2ib5 failed: rc = -16 Sep 03 15:17:29 oak-md1-s2 kernel: LustreError: Skipped 41 previous similar messages Sep 03 15:17:54 oak-md1-s2 kernel: LustreError: 11-0: oak-OST0001-osc-MDT0000: operation ost_connect to node 10.0.2.102@o2ib5 failed: rc = -16 Sep 03 15:17:54 oak-md1-s2 kernel: LustreError: Skipped 95 previous similar messages Sep 03 15:18:19 oak-md1-s2 kernel: LustreError: 11-0: oak-OST000f-osc-MDT0000: operation ost_connect to node 10.0.2.102@o2ib5 failed: rc = -16 Sep 03 15:18:19 oak-md1-s2 kernel: LustreError: Skipped 48 previous similar messages Sep 03 15:19:09 oak-md1-s2 kernel: LustreError: 11-0: oak-OST000b-osc-MDT0000: operation ost_connect to node 10.0.2.102@o2ib5 failed: rc = -16 Sep 03 15:19:09 oak-md1-s2 kernel: LustreError: Skipped 141 previous similar messages Sep 03 15:19:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Will be in recovery for at least 5:00, or until 71 clients reconnect Sep 03 15:19:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 5:01 Sep 03 15:19:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 4:36 Sep 03 15:20:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 4:11 Sep 03 15:20:24 oak-md1-s2 kernel: LustreError: 11-0: oak-OST0004-osc-MDT0000: operation ost_connect to node 10.0.2.101@o2ib5 failed: rc = -16 Sep 03 15:20:24 oak-md1-s2 kernel: LustreError: Skipped 231 previous similar messages Sep 03 15:20:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 3:46 Sep 03 15:20:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 3:21 Sep 03 15:21:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 2:56 Sep 03 15:21:18 oak-md1-s2 kernel: LustreError: 4274:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88102cfd9850 x1610626266860224/t0(0) o601->oak-MDT0000-lwp-OST0046_UUID@10.0.2.105@o2ib5:514/0 lens 336/0 e 0 to 0 dl 1536013284 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:21:18 oak-md1-s2 kernel: LustreError: 4274:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 9 previous similar messages Sep 03 15:21:18 oak-md1-s2 kernel: LustreError: 4240:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88202cfe9b00 x1610626266860416/t0(0) o601->oak-MDT0000-lwp-OST005d_UUID@10.0.2.106@o2ib5:514/0 lens 336/0 e 0 to 0 dl 1536013284 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:21:18 oak-md1-s2 kernel: LustreError: 4240:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 19 previous similar messages Sep 03 15:21:20 oak-md1-s2 kernel: LustreError: 4243:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88202cfe8c00 x1610626266860672/t0(0) o601->oak-MDT0000-lwp-OST0059_UUID@10.0.2.106@o2ib5:516/0 lens 336/0 e 0 to 0 dl 1536013286 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:21:20 oak-md1-s2 kernel: LustreError: 4243:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 28 previous similar messages Sep 03 15:21:22 oak-md1-s2 kernel: LustreError: 4244:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff882015cf4050 x1610626266861216/t0(0) o601->oak-MDT0000-lwp-OST0049_UUID@10.0.2.106@o2ib5:518/0 lens 336/0 e 0 to 0 dl 1536013288 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:21:22 oak-md1-s2 kernel: LustreError: 4244:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 49 previous similar messages Sep 03 15:21:26 oak-md1-s2 kernel: LustreError: 4265:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88102cfddc50 x1610626266862128/t0(0) o601->oak-MDT0000-lwp-OST0032_UUID@10.0.2.105@o2ib5:522/0 lens 336/0 e 0 to 0 dl 1536013292 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:21:26 oak-md1-s2 kernel: LustreError: 4265:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 106 previous similar messages Sep 03 15:21:35 oak-md1-s2 kernel: LustreError: 4250:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88200b8a3f00 x1610626220727280/t0(0) o601->oak-MDT0000-lwp-OST001c_UUID@10.0.2.101@o2ib5:531/0 lens 336/0 e 0 to 0 dl 1536013301 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:21:35 oak-md1-s2 kernel: LustreError: 4250:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 209 previous similar messages Sep 03 15:21:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 2:31 Sep 03 15:21:51 oak-md1-s2 kernel: LustreError: 4287:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88200b8a2100 x1610626220730448/t0(0) o601->oak-MDT0000-lwp-OST0002_UUID@10.0.2.101@o2ib5:547/0 lens 336/0 e 0 to 0 dl 1536013317 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:21:51 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 2 seconds Sep 03 15:21:51 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 3 previous similar messages Sep 03 15:21:51 oak-md1-s2 kernel: LustreError: 4287:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 323 previous similar messages Sep 03 15:22:23 oak-md1-s2 kernel: LustreError: 4265:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88103d57b600 x1610626266865648/t0(0) o601->oak-MDT0000-lwp-OST0054_UUID@10.0.2.105@o2ib5:579/0 lens 336/0 e 0 to 0 dl 1536013349 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:22:23 oak-md1-s2 kernel: LustreError: 4265:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 176 previous similar messages Sep 03 15:22:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:41 Sep 03 15:22:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 15:23:28 oak-md1-s2 kernel: LustreError: 4246:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff882015cf7050 x1610626266870592/t0(0) o601->oak-MDT0000-lwp-OST004f_UUID@10.0.2.106@o2ib5:643/0 lens 336/0 e 0 to 0 dl 1536013413 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 03 15:23:28 oak-md1-s2 kernel: LustreError: 4246:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 867 previous similar messages Sep 03 15:23:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 17037205-59df-58c0-2551-cc5c9765abda(at 10.0.2.220@o2ib5), waiting for 71 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:26 Sep 03 15:23:46 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 03 15:24:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: recovery is timed out, evict stale exports Sep 03 15:24:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: disconnecting 71 stale clients Sep 03 15:24:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery over after 5:02, of 71 clients 0 recovered and 71 were evicted. Sep 03 15:24:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.0.2.220@o2ib5) Sep 03 15:25:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Sep 03 15:34:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 824977c0-2cb5-3750-c1c7-1cab28e50124 (at 10.12.4.20@o2ib) Sep 03 15:41:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ae7961f1-c27d-d378-44be-730fb7c7cbd3 (at 10.12.4.65@o2ib) Sep 03 15:43:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 316d4d97-f921-6627-ff1c-f0675198a07c (at 10.0.2.225@o2ib5) Sep 03 15:43:28 oak-md1-s2 kernel: Lustre: Skipped 65 previous similar messages Sep 03 15:50:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b86cd386-a2f2-b44a-c0eb-72da3b280f32 (at 10.0.2.243@o2ib5) Sep 03 15:50:17 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 03 16:18:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 16:18:57 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 03 16:25:20 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.240@o2ib5 version 12/12 incarnation 1536014815370529/1536017093283876 Sep 03 16:25:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6f54a780-216a-889a-53c7-12f6d371476f (at 10.0.2.240@o2ib5) Sep 03 16:31:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) Sep 03 16:31:34 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 03 16:36:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 684d53b4-3df6-9d5b-0506-1d1fcbfe8402 (at 10.0.2.251@o2ib5) Sep 03 16:46:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 29894e65-6342-63e7-25d3-e58d38da245b (at 10.8.6.34@o2ib6) Sep 03 16:46:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 03 16:56:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 29894e65-6342-63e7-25d3-e58d38da245b (at 10.8.6.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810392fdc00, cur 1536019019 expire 1536018869 last 1536018792 Sep 03 17:02:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5db97283-e9e4-ad12-d4da-1d3c81838f6c (at 10.0.2.246@o2ib5) Sep 03 17:02:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 17:17:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6429f5e-e834-a3b3-1935-f62bb506f90c (at 10.12.4.24@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff88103890b800, cur 1536020257 expire 1536020107 last 1536020029 Sep 03 17:18:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6429f5e-e834-a3b3-1935-f62bb506f90c (at 10.12.4.24@o2ib) Sep 03 17:19:50 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.243@o2ib5 version 12/12 incarnation 1536014989902452/1536020364361230 Sep 03 17:20:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 17:21:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 17:21:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 17:21:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b86cd386-a2f2-b44a-c0eb-72da3b280f32 (at 10.0.2.243@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200d3fdc00, cur 1536020494 expire 1536020344 last 1536020267 Sep 03 17:21:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 17:41:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 17:41:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 17:41:55 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 03 17:42:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 17:43:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 17:43:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 17:43:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 17:43:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 17:46:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 29894e65-6342-63e7-25d3-e58d38da245b (at 10.8.6.34@o2ib6) Sep 03 17:46:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 17:59:56 oak-md1-s2 kernel: LNet: 3780:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 18:00:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:00:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 18:02:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:05:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:08:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:09:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:09:26 oak-md1-s2 kernel: LNet: 3782:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 18:10:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:10:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 18:10:50 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 03 18:33:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:33:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 18:33:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 18:33:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 18:35:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 18:35:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 18:35:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 18:52:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3e6b8413-a309-e923-5342-2526cadfe821 (at 10.8.0.66@o2ib6) Sep 03 19:29:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e6af7493-5c77-067f-6cee-37f589a3ba62 (at 10.8.2.19@o2ib6) Sep 03 19:29:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 19:29:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 29230e55-0d9a-81e3-19d4-040fc566af73 (at 10.8.1.16@o2ib6) Sep 03 19:29:40 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 03 19:29:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 735293f9-7966-d6f0-f715-d76594f7d8c9 (at 10.8.1.34@o2ib6) Sep 03 19:29:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 03 19:30:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6287b08e-7a2b-d21f-9862-c53903ea650f (at 10.8.1.33@o2ib6) Sep 03 19:30:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 03 19:30:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 91c0a371-2eb7-6af5-f927-a00df13ee31f (at 10.8.2.6@o2ib6) Sep 03 19:30:20 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 03 19:31:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4d14dfe0-6c84-38d9-8a1e-663195f9cb97 (at 10.8.2.29@o2ib6) Sep 03 19:31:00 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Sep 03 19:32:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4b119af-2df3-64a1-a0dc-9e0c140c1823 (at 10.8.4.12@o2ib6) Sep 03 19:32:15 oak-md1-s2 kernel: Lustre: Skipped 39 previous similar messages Sep 03 19:34:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 78702acc-4024-91e2-059f-0ec7adbffa87 (at 10.8.15.9@o2ib6) Sep 03 19:34:51 oak-md1-s2 kernel: Lustre: Skipped 42 previous similar messages Sep 03 19:40:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ddc62b0c-7246-3040-e182-15771ad9df6e (at 10.8.10.25@o2ib6) Sep 03 19:40:05 oak-md1-s2 kernel: Lustre: Skipped 127 previous similar messages Sep 03 19:50:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cde12fa6-db73-c505-158a-513511bfbb56 (at 10.9.104.68@o2ib4) Sep 03 19:50:05 oak-md1-s2 kernel: Lustre: Skipped 466 previous similar messages Sep 03 19:52:20 oak-md1-s2 kernel: LNet: 3780:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 19:52:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 19:53:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 19:56:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 19:58:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 20:00:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 20:00:02 oak-md1-s2 kernel: LNet: 3781:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 20:01:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 20:01:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 20:01:35 oak-md1-s2 kernel: Lustre: Skipped 165 previous similar messages Sep 03 20:03:49 oak-md1-s2 kernel: perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Sep 03 20:32:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9be6edfa-ebde-1d5d-f312-2249d8fda40b (at 10.8.2.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808cf36c400, cur 1536031925 expire 1536031775 last 1536031698 Sep 03 20:49:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7dc431a-f049-e1b5-21a8-c5ef3afa8302 (at 10.8.4.27@o2ib6) Sep 03 20:52:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b69140d-a83b-c287-c38c-8a22abebae36 (at 10.9.0.62@o2ib4) Sep 03 20:56:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48ec7f74-5b7c-f0ad-fb35-6041a89f83d3 (at 10.8.4.16@o2ib6) Sep 03 21:12:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 03 21:35:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7b76ea0a-8a2e-0b62-84dd-0e128616b2f7 (at 10.8.21.1@o2ib6) Sep 03 21:35:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 21:37:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c2205683-0c30-1b2d-149a-2ac48caff36c (at 10.8.26.11@o2ib6) Sep 03 21:40:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 21:40:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 21:40:36 oak-md1-s2 kernel: Lustre: Skipped 387 previous similar messages Sep 03 21:44:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b067ccdd-27cb-ae5c-27d9-df62cb606375 (at 10.8.4.19@o2ib6) Sep 03 21:55:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bba14768-336e-292a-9905-5121b3694a55 (at 10.8.0.67@o2ib6) Sep 03 21:59:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3eac4d27-6938-5bd3-6cf0-77bbdf595f11 (at 10.9.0.61@o2ib4) Sep 03 22:09:40 oak-md1-s2 kernel: LNet: 3781:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 22:10:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 22:10:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 22:11:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 22:11:26 oak-md1-s2 kernel: LNet: 3781:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 22:11:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 22:13:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 22:23:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bba14768-336e-292a-9905-5121b3694a55 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d0c34b000, cur 1536038587 expire 1536038437 last 1536038360 Sep 03 22:23:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 22:33:19 oak-md1-s2 kernel: perf: interrupt took too long (3136 > 3131), lowering kernel.perf_event_max_sample_rate to 63000 Sep 03 22:35:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6b38b312-8284-0d4b-ebb6-3c34f1c9425b (at 10.8.18.22@o2ib6) Sep 03 22:35:22 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 03 22:36:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d1718f02-45a3-37a9-c4b8-825039be68b6 (at 10.8.8.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8ca9c000, cur 1536039383 expire 1536039233 last 1536039156 Sep 03 22:36:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 22:39:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 03 22:39:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 03 22:39:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c607fd68-5957-21e5-e9c2-2083dc94ed9d (at 10.8.3.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102abeb400, cur 1536039591 expire 1536039441 last 1536039364 Sep 03 22:39:51 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 03 22:49:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 271597b8-7305-5b82-15d7-58d2b2a4dd26 (at 10.9.104.3@o2ib4) Sep 03 22:53:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6616415b-e8c5-8cbd-0139-1aa6866f897e (at 10.9.114.12@o2ib4) Sep 03 22:53:01 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Sep 03 22:53:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b69140d-a83b-c287-c38c-8a22abebae36 (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b743400, cur 1536040408 expire 1536040258 last 1536040181 Sep 03 22:59:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 47cf49c5-0515-7f67-da5c-d23e377ac991 (at 10.9.0.81@o2ib4) Sep 03 22:59:06 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 03 23:09:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bced131c-6b24-fe29-db28-e98ba8348efb (at 10.8.0.82@o2ib6) Sep 03 23:09:07 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 03 23:15:14 oak-md1-s2 kernel: LNet: 3782:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 23:16:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:19:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:19:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 23:19:20 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 03 23:22:03 oak-md1-s2 kernel: LNet: 3779:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 23:22:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:25:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:25:36 oak-md1-s2 kernel: LNet: 3781:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 03 23:27:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:28:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e6b8413-a309-e923-5342-2526cadfe821 (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808cf36ac00, cur 1536042509 expire 1536042359 last 1536042282 Sep 03 23:28:29 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 03 23:29:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:29:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 03 23:29:40 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 03 23:32:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:34:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 03 23:42:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e11c5608-046e-951d-1d45-f86ff3fdb5c2 (at 10.8.6.20@o2ib6) Sep 03 23:42:38 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 03 23:54:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e8f82620-e480-b4c7-f7a4-247bcbefc4a2 (at 10.8.0.68@o2ib6) Sep 03 23:54:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 03 23:57:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d342f90e-597c-3ec6-8e55-42583e98e20d (at 10.9.107.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a53c00, cur 1536044250 expire 1536044100 last 1536044023 Sep 03 23:57:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 04 00:24:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 00:24:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 00:24:34 oak-md1-s2 kernel: LNet: 3779:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 04 00:25:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 00:25:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 00:29:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 29894e65-6342-63e7-25d3-e58d38da245b (at 10.8.6.34@o2ib6) Sep 04 00:29:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 00:54:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 00:54:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 00:54:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 00:56:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 00:56:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 00:57:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2ca000, cur 1536047825 expire 1536047675 last 1536047598 Sep 04 00:59:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 00:59:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 00:59:13 oak-md1-s2 kernel: LNet: 3780:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 04 01:02:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 01:02:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 01:04:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 01:07:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 01:07:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 01:07:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 02:08:16 oak-md1-s2 kernel: perf: interrupt took too long (3923 > 3920), lowering kernel.perf_event_max_sample_rate to 50000 Sep 04 05:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 05:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 05:57:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 05:57:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 05:58:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 05:58:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 05:59:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 05:59:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 06:01:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 06:01:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 06:01:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 06:02:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 06:02:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 06:02:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 06:32:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102dfee000, cur 1536067935 expire 1536067785 last 1536067708 Sep 04 06:32:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aa36fc16-505e-a9e9-38b1-6d2cc8b192a1 (at 10.8.21.19@o2ib6) Sep 04 06:33:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 04 06:36:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to db0130ba-6755-04c2-1981-5eeca0fbcd17 (at 10.9.113.15@o2ib4) Sep 04 06:36:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48ec7f74-5b7c-f0ad-fb35-6041a89f83d3 (at 10.8.4.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200be4dc00, cur 1536068165 expire 1536068015 last 1536067938 Sep 04 06:36:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 04 06:47:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 04 06:48:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 04 06:49:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6d4765fe-3f90-54e3-5726-62369fd2bc50 (at 10.8.19.5@o2ib6) Sep 04 06:51:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6f11ebc-0bd6-ad8b-390a-e041d9836de8 (at 10.9.115.6@o2ib4) Sep 04 06:52:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6616415b-e8c5-8cbd-0139-1aa6866f897e (at 10.9.114.12@o2ib4) Sep 04 06:52:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 07:33:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bba14768-336e-292a-9905-5121b3694a55 (at 10.8.0.67@o2ib6) Sep 04 07:39:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 04 07:51:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3e6b8413-a309-e923-5342-2526cadfe821 (at 10.8.0.66@o2ib6) Sep 04 07:51:39 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 04 08:06:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43038ec1-e22e-886f-2a71-7f475d6cef88 (at 10.8.20.24@o2ib6) Sep 04 08:06:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 08:06:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7ee64c2f-c137-ef43-af31-6ea0dd54fcce (at 10.8.20.7@o2ib6) Sep 04 08:07:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27bac3ac-d257-a563-4125-c911a485eb0b (at 10.9.101.60@o2ib4) Sep 04 08:09:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c607fd68-5957-21e5-e9c2-2083dc94ed9d (at 10.8.3.17@o2ib6) Sep 04 08:09:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d1718f02-45a3-37a9-c4b8-825039be68b6 (at 10.8.8.22@o2ib6) Sep 04 08:10:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4be8789e-d2ce-182c-849d-c564d8af1819 (at 10.8.11.15@o2ib6) Sep 04 08:10:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ebcc7456-5e8f-e275-1e09-9882443932f1 (at 10.8.3.32@o2ib6) Sep 04 08:10:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 08:11:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca2911d7-e15b-8887-321a-5c3bb16e7abb (at 10.8.10.22@o2ib6) Sep 04 08:11:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 08:16:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 56c644c5-922f-0c6b-a7ec-8f493c7e6a52 (at 10.8.6.25@o2ib6) Sep 04 08:16:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 08:47:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 430f70a6-735b-dec3-13cf-4f919b0b9772 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b61436000, cur 1536076069 expire 1536075919 last 1536075842 Sep 04 08:47:49 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 04 09:02:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200d3fdc00, cur 1536076925 expire 1536076775 last 1536076698 Sep 04 09:02:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:12:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f6bd8f3-da25-cf60-85cc-1fc5dd0e6bbd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fbc384800, cur 1536077568 expire 1536077418 last 1536077341 Sep 04 09:16:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:20:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:21:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 95af7bf5-3f17-dd3b-2dac-8177cb35b3c4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88022f041800, cur 1536078071 expire 1536077921 last 1536077844 Sep 04 09:25:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:25:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client af9cb71d-f862-279d-58b5-f726480caa17 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801d9128400, cur 1536078333 expire 1536078183 last 1536078106 Sep 04 09:28:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a45b77000, cur 1536078525 expire 1536078375 last 1536078298 Sep 04 09:29:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:30:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc8888c9-ea4e-1ebb-f9e4-fd1c5e7cf5eb (at 10.8.18.35@o2ib6) in 215 seconds. I think it's dead, and I am evicting it. exp ffff880380a47400, cur 1536078601 expire 1536078451 last 1536078386 Sep 04 09:35:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da2a1d14-97b6-5bae-5dab-e54972098fed (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff6e71000, cur 1536078955 expire 1536078805 last 1536078728 Sep 04 09:40:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:49:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 188d2207-fb78-5ede-dda0-bfc5890eb8e3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0c925800, cur 1536079758 expire 1536079608 last 1536079531 Sep 04 09:51:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:56:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 09:56:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5cf8e8a9-f3da-4259-c320-f637d2bc629b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88094c4b8000, cur 1536080199 expire 1536080049 last 1536079972 Sep 04 09:57:52 oak-md1-s2 kernel: list passed to list_sort() too long for efficiency Sep 04 09:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Sep 04 10:00:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cbfa98ba-b789-f569-1977-7b670a2f17c7 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88057bcda400, cur 1536080402 expire 1536080252 last 1536080175 Sep 04 10:01:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14c7ca7b-e54a-24f5-d845-29bab5f1ef8a (at 10.0.2.249@o2ib5) Sep 04 10:01:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 96d66914-1a95-c9fc-e081-f785738e73e3 (at 10.8.18.35@o2ib6) in 216 seconds. I think it's dead, and I am evicting it. exp ffff880c46161800, cur 1536080478 expire 1536080328 last 1536080262 Sep 04 10:01:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 04 10:10:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 04 10:54:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e11c5608-046e-951d-1d45-f86ff3fdb5c2 (at 10.8.6.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803cb635800, cur 1536083686 expire 1536083536 last 1536083459 Sep 04 11:03:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 04 11:15:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0b334521-e8aa-58c6-0fd8-449db0274f2c (at 10.8.17.30@o2ib6) Sep 04 11:23:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6cdfa4a1-0eb5-fc32-84dc-5732f06d5d85 (at 10.8.4.7@o2ib6) Sep 04 11:24:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b067ccdd-27cb-ae5c-27d9-df62cb606375 (at 10.8.4.19@o2ib6) Sep 04 11:32:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e11c5608-046e-951d-1d45-f86ff3fdb5c2 (at 10.8.6.20@o2ib6) Sep 04 11:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8745a439-5fc8-4945-3730-a23e61343857 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805e58a6000, cur 1536086087 expire 1536085937 last 1536085860 Sep 04 11:34:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 11:35:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 04 11:38:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7dc431a-f049-e1b5-21a8-c5ef3afa8302 (at 10.8.4.27@o2ib6) Sep 04 11:47:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e31134b5-e472-9719-70da-4bce05f885b0 (at 10.8.14.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca4d400, cur 1536086850 expire 1536086700 last 1536086623 Sep 04 12:11:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 04 12:11:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 04 12:11:33 oak-md1-s2 kernel: LustreError: 4300:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.0.2.3@o2ib5) returned error from blocking AST (req@ffff880a60190300 x1610626357373232 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880c4e8eca00/0x6c00c3d010481cf7 lrc: 4/0,0 mode: PR/PR res: [0x200017094:0x4:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.0.2.3@o2ib5 remote: 0xafd8851dd9569e4b expref: 3576586 pid: 10659 timeout: 4373386026 lvb_type: 0 Sep 04 12:11:33 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.0.2.3@o2ib5 was evicted due to a lock blocking callback time out: rc -107 Sep 04 12:11:33 oak-md1-s2 kernel: LustreError: 10848:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.0.2.3@o2ib5 arrived at 1536088293 with bad export cookie 7782435437524808406 Sep 04 12:11:33 oak-md1-s2 kernel: LustreError: 10848:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) Skipped 3969 previous similar messages Sep 04 12:11:38 oak-md1-s2 kernel: LustreError: 10650:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881c486cb900 x1610626357375024/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:11:54 oak-md1-s2 kernel: LustreError: 10485:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880054506f00 x1610626357379952/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:12:12 oak-md1-s2 kernel: LustreError: 4256:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800577def00 x1610626357386560/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:12:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4966203-761f-6dc5-9911-bf9e91bbb908 (at 10.8.13.24@o2ib6) Sep 04 12:12:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 12:12:16 oak-md1-s2 kernel: LustreError: 10556:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88001cfab000 x1610626357389088/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:12:35 oak-md1-s2 kernel: LustreError: 10581:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004790a400 x1610626357397488/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:12:35 oak-md1-s2 kernel: LustreError: 10581:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Sep 04 12:13:12 oak-md1-s2 kernel: LustreError: 10556:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88001cfa8f00 x1610626357409392/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:13:39 oak-md1-s2 kernel: LustreError: 10569:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881b7f6af800 x1610626357421728/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:13:39 oak-md1-s2 kernel: LustreError: 10569:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Sep 04 12:14:53 oak-md1-s2 kernel: LNet: Service thread pid 4300 was inactive for 200.34s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 04 12:14:53 oak-md1-s2 kernel: Pid: 4300, comm: mdt00_008 Sep 04 12:14:53 oak-md1-s2 kernel: Sep 04 12:14:53 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 04 12:14:53 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 04 12:14:53 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 04 12:14:53 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 04 12:14:53 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] mdt_reint_object_lock+0x20/0x60 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] mdt_reint_unlink+0x842/0x11d0 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? mdt_root_squash+0x21/0x430 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] mdt_reint+0x67/0x140 [mdt] Sep 04 12:14:53 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 04 12:14:53 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 04 12:14:53 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 04 12:14:53 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 04 12:14:53 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:14:53 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 04 12:14:53 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:14:53 oak-md1-s2 kernel: Sep 04 12:14:53 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088493.4300 Sep 04 12:14:57 oak-md1-s2 kernel: LNet: Service thread pid 4546 was inactive for 200.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 04 12:14:57 oak-md1-s2 kernel: Pid: 4546, comm: mdt00_012 Sep 04 12:14:57 oak-md1-s2 kernel: Sep 04 12:14:57 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 04 12:14:57 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 04 12:14:57 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 04 12:14:57 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 04 12:14:57 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Sep 04 12:14:57 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 04 12:14:57 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? osd_fld_lookup+0x48/0xd0 [osd_ldiskfs] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 04 12:14:58 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? lu_object_find+0x16/0x20 [obdclass] Sep 04 12:14:58 oak-md1-s2 kernel: [] mdt_getattr_name_lock+0x8a4/0x1900 [mdt] Sep 04 12:14:58 oak-md1-s2 kernel: [] mdt_intent_getattr+0x2b0/0x480 [mdt] Sep 04 12:14:58 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Sep 04 12:14:58 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 04 12:14:58 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 04 12:14:58 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 04 12:14:58 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 04 12:14:58 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:14:58 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 04 12:14:58 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:14:58 oak-md1-s2 kernel: Sep 04 12:14:58 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088498.4546 Sep 04 12:15:12 oak-md1-s2 kernel: LNet: Service thread pid 4300 completed after 219.76s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Sep 04 12:15:12 oak-md1-s2 kernel: LNet: Skipped 1 previous similar message Sep 04 12:15:14 oak-md1-s2 kernel: LNet: Service thread pid 10538 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 04 12:15:14 oak-md1-s2 kernel: Pid: 10538, comm: mdt01_043 Sep 04 12:15:14 oak-md1-s2 kernel: Sep 04 12:15:14 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 04 12:15:14 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 04 12:15:14 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_rename_or_migrate.isra.39+0x67c/0x860 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_rename+0x13/0x20 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint+0x67/0x140 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 04 12:15:14 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:15:14 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:15:14 oak-md1-s2 kernel: Sep 04 12:15:14 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088514.10538 Sep 04 12:15:14 oak-md1-s2 kernel: Pid: 10485, comm: mdt00_023 Sep 04 12:15:14 oak-md1-s2 kernel: Sep 04 12:15:14 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 04 12:15:14 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 04 12:15:14 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_object_lock+0x20/0x60 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_rename_internal.isra.36+0x1a6d/0x20c0 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_rename_or_migrate.isra.39+0x19b/0x860 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_rename+0x13/0x20 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] mdt_reint+0x67/0x140 [mdt] Sep 04 12:15:14 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 04 12:15:14 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 04 12:15:14 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:15:14 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 04 12:15:14 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:15:14 oak-md1-s2 kernel: Sep 04 12:15:15 oak-md1-s2 kernel: Pid: 10567, comm: mdt00_069 Sep 04 12:15:15 oak-md1-s2 kernel: Sep 04 12:15:15 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 04 12:15:15 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 04 12:15:15 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 04 12:15:15 oak-md1-s2 kernel: [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 04 12:15:15 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] mdt_reint_rename_or_migrate.isra.39+0x67c/0x860 [mdt] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] mdt_reint_rename+0x13/0x20 [mdt] Sep 04 12:15:15 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 04 12:15:15 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 04 12:15:15 oak-md1-s2 kernel: [] mdt_reint+0x67/0x140 [mdt] Sep 04 12:15:15 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 04 12:15:15 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 04 12:15:15 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 04 12:15:15 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 04 12:15:15 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:15:15 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 04 12:15:15 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 04 12:15:15 oak-md1-s2 kernel: Sep 04 12:15:15 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088515.10567 Sep 04 12:15:16 oak-md1-s2 kernel: LNet: Service thread pid 10633 was inactive for 200.35s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Sep 04 12:15:16 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088516.10633 Sep 04 12:15:17 oak-md1-s2 kernel: LNet: Service thread pid 10549 was inactive for 200.30s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Sep 04 12:15:17 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088517.10549 Sep 04 12:15:38 oak-md1-s2 kernel: LNet: Service thread pid 10572 was inactive for 200.43s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Sep 04 12:15:38 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088538.10572 Sep 04 12:15:55 oak-md1-s2 kernel: LNet: Service thread pid 10581 was inactive for 200.31s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Sep 04 12:15:55 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088555.10581 Sep 04 12:16:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5d4391d4-9b99-8325-5171-22df7be894e2 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803a97b5800, cur 1536088563 expire 1536088413 last 1536088336 Sep 04 12:16:07 oak-md1-s2 kernel: LNet: Service thread pid 10547 was inactive for 200.06s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Sep 04 12:16:07 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088567.10547 Sep 04 12:16:54 oak-md1-s2 kernel: LustreError: 10485:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1536088314, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff8805f100a200/0x6c00c3d0109e2c35 lrc: 3/0,1 mode: --/EX res: [0x200017068:0x11b:0x0].0x0 bits 0x3 rrc: 4 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 10485 timeout: 0 lvb_type: 0 Sep 04 12:16:54 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536088614.10485 Sep 04 12:16:55 oak-md1-s2 kernel: LustreError: 10567:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1536088315, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff881fec55ac00/0x6c00c3d0109e7fd3 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2 rrc: 9 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 10567 timeout: 0 lvb_type: 0 Sep 04 12:16:55 oak-md1-s2 kernel: LustreError: 10567:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message Sep 04 12:16:57 oak-md1-s2 kernel: LustreError: 10549:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1536088317, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880fb1f04800/0x6c00c3d0109f4887 lrc: 3/0,1 mode: --/EX res: [0x200000004:0x1:0x0].0x0 bits 0x2 rrc: 9 type: IBT flags: 0x40210400000020 nid: local remote: 0x0 expref: -99 pid: 10549 timeout: 0 lvb_type: 0 Sep 04 12:16:57 oak-md1-s2 kernel: LustreError: 10549:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message Sep 04 12:17:17 oak-md1-s2 kernel: LustreError: 10572:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1536088337, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880aa68ac000/0x6c00c3d010a8e84f lrc: 3/1,0 mode: --/PR res: [0x200017068:0x11b:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10572 timeout: 0 lvb_type: 0 Sep 04 12:17:19 oak-md1-s2 kernel: LNet: Service thread pid 10581 completed after 284.44s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Sep 04 12:17:23 oak-md1-s2 kernel: LNet: Service thread pid 10485 completed after 328.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Sep 04 12:17:23 oak-md1-s2 kernel: LNet: Skipped 3 previous similar messages Sep 04 12:17:23 oak-md1-s2 kernel: LustreError: 10672:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880015d99b00 x1610626357498352/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 04 12:19:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3910199a-80eb-8b7e-a4ce-25c88dcaaf65 (at 10.8.23.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809717dfc00, cur 1536088772 expire 1536088622 last 1536088545 Sep 04 12:26:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bccf629c-900e-c4a9-171a-d70129138265 (at 10.8.14.1@o2ib6) Sep 04 12:26:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 12:26:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4f8ff9b5-3c9f-09af-6ef9-a813290ae6d7 (at 10.8.14.8@o2ib6) Sep 04 12:26:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 12:41:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4a9b9ea1-3786-1d90-2c15-70a68834d78c (at 10.8.3.1@o2ib6) Sep 04 12:41:37 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 04 12:43:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3910199a-80eb-8b7e-a4ce-25c88dcaaf65 (at 10.8.23.24@o2ib6) Sep 04 12:54:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fc4b5271-be0d-0e0a-a2d7-5e5e4e3f2320 (at 10.8.14.2@o2ib6) Sep 04 12:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e51e754b-5642-16b9-274d-64675feb1b56 (at 10.8.23.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063fd8000, cur 1536091112 expire 1536090962 last 1536090885 Sep 04 12:59:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9bb3e127-2315-6445-6831-bc9961a2a63c (at 10.8.14.2@o2ib6) in 154 seconds. I think it's dead, and I am evicting it. exp ffff880ff1e68c00, cur 1536091188 expire 1536091038 last 1536091034 Sep 04 13:06:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3910199a-80eb-8b7e-a4ce-25c88dcaaf65 (at 10.8.23.24@o2ib6) Sep 04 13:29:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 35909673-1b83-fd46-7a8d-9f8adfebd493 (at 10.8.23.34@o2ib6) Sep 04 13:39:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3449ec0-e6fd-05c9-1da1-faeeaf95c643 (at 10.8.12.11@o2ib6) Sep 04 13:51:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 31c4ea76-8d98-76b3-0d2a-f267092359fe (at 10.9.101.72@o2ib4) Sep 04 14:05:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 15ec7c10-56ee-ec2a-0465-ab8d719c7044 (at 10.9.105.25@o2ib4) Sep 04 14:06:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d864271d-68ba-37ce-9609-2c6659593006 (at 10.9.105.28@o2ib4) Sep 04 14:20:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2c181141-00e9-9dd2-49c2-18dde8e229ef (at 10.9.102.47@o2ib4) Sep 04 14:22:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 58bcf75f-4fde-4a3f-5845-4b3c5e7fe6e9 (at 10.9.104.30@o2ib4) Sep 04 14:34:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d342f90e-597c-3ec6-8e55-42583e98e20d (at 10.9.107.31@o2ib4) Sep 04 14:43:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 430f70a6-735b-dec3-13cf-4f919b0b9772 (at 10.8.9.9@o2ib6) Sep 04 14:46:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b3642d76-692a-4f13-7dd6-60d71e2ccc49 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff1e6ac00, cur 1536097580 expire 1536097430 last 1536097353 Sep 04 14:58:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 04 15:05:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 04 15:24:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 357c9327-5943-7b6b-bf71-96f316a93e47 (at 10.0.2.247@o2ib5) Sep 04 15:27:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882038609000, cur 1536100050 expire 1536099900 last 1536099823 Sep 04 15:30:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 04 15:42:14 oak-md1-s2 kernel: Lustre: DEBUG MARKER: Tue Sep 4 15:42:14 2018 Sep 04 15:43:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Sep 04 15:49:30 oak-md1-s2 kernel: LNet: 464:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.250@o2ib5 version 12/12 incarnation 1536018853952259/1536101345633341 Sep 04 15:49:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 964ef87c-7200-84ec-a437-541be4d22b25 (at 10.0.2.250@o2ib5) Sep 04 15:51:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 964ef87c-7200-84ec-a437-541be4d22b25 (at 10.0.2.250@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c633400, cur 1536101483 expire 1536101333 last 1536101256 Sep 04 15:51:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 04 15:55:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 167059a8-887d-f282-1560-0fbcdd188dab (at 10.8.13.23@o2ib6) Sep 04 15:56:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fc4b5271-be0d-0e0a-a2d7-5e5e4e3f2320 (at 10.8.14.2@o2ib6) Sep 04 16:02:30 oak-md1-s2 kernel: Lustre: DEBUG MARKER: Tue Sep 4 16:02:30 2018 Sep 04 16:03:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe535c800, cur 1536102214 expire 1536102064 last 1536101987 Sep 04 16:06:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d049b869-d728-0a9c-64a8-4c2cb1a8428f (at 10.8.3.7@o2ib6) Sep 04 16:15:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d342f90e-597c-3ec6-8e55-42583e98e20d (at 10.9.107.31@o2ib4) Sep 04 16:24:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Sep 04 16:48:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ff2d024-1560-78b9-57cc-b9b2f8fdceed (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f4c76a400, cur 1536104905 expire 1536104755 last 1536104678 Sep 04 17:08:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Sep 04 17:15:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882038609800, cur 1536106500 expire 1536106350 last 1536106273 Sep 04 17:18:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 690928dc-349a-ea89-27f4-fef5467fee41 (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880568ea0000, cur 1536106729 expire 1536106579 last 1536106502 Sep 04 17:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Sep 04 17:39:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Sep 04 18:23:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9159d552-9efc-2aa5-0a82-ea310c396d94 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880024a36000, cur 1536110597 expire 1536110447 last 1536110370 Sep 04 18:48:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 04 19:15:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 04 19:16:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df3e67a6-9202-0563-f946-489413a34c48 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2de67000, cur 1536113769 expire 1536113619 last 1536113542 Sep 04 20:23:13 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 04 21:03:16 oak-md1-s2 kernel: perf: interrupt took too long (4913 > 4903), lowering kernel.perf_event_max_sample_rate to 40000 Sep 04 22:35:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94a571c6-b72d-ed1a-cbba-42672498fdc4 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d0cc3e000, cur 1536125733 expire 1536125583 last 1536125506 Sep 05 00:01:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e58be353-6bc0-1c69-b54d-1db7c01f7d82 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8698000, cur 1536130900 expire 1536130750 last 1536130673 Sep 05 00:19:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e58be353-6bc0-1c69-b54d-1db7c01f7d82 (at 10.9.112.17@o2ib4) Sep 05 02:28:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a129aacc-d5a2-8ce7-a40c-10a1ab338c77 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805879b2c00, cur 1536139690 expire 1536139540 last 1536139463 Sep 05 02:46:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e58be353-6bc0-1c69-b54d-1db7c01f7d82 (at 10.9.112.17@o2ib4) Sep 05 05:10:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a773000e-59f7-f84c-722c-847586d6d9dd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea1c00, cur 1536149427 expire 1536149277 last 1536149200 Sep 05 05:29:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 05 06:17:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 05 06:19:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6200201-bb6d-dda1-9ced-505fffc90c62 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5968bc00, cur 1536153585 expire 1536153435 last 1536153358 Sep 05 09:58:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 07da5e87-6a86-7fdf-7693-c2ca350af724 (at 10.0.2.223@o2ib5) Sep 05 10:10:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809717db400, cur 1536167406 expire 1536167256 last 1536167179 Sep 05 10:47:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 05 10:48:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc8abc27-55d8-6c1f-9da5-8d606a7621ab (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c5a1800, cur 1536169717 expire 1536169567 last 1536169490 Sep 05 10:56:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2db31ebf-ddc0-492d-4b67-eb0588485957 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa010c00, cur 1536170196 expire 1536170046 last 1536169969 Sep 05 11:10:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 05 11:11:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5db97283-e9e4-ad12-d4da-1d3c81838f6c (at 10.0.2.246@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882009856800, cur 1536171107 expire 1536170957 last 1536170880 Sep 05 11:14:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 05 11:18:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1c987093-1325-6629-623f-7a472b8876c7 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d75d4bc00, cur 1536171501 expire 1536171351 last 1536171274 Sep 05 11:32:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d049b869-d728-0a9c-64a8-4c2cb1a8428f (at 10.8.3.7@o2ib6) Sep 05 11:36:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 05 12:02:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e4ae2c71-7024-594c-6d81-f7f14ac78fd2 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b415dd400, cur 1536174156 expire 1536174006 last 1536173929 Sep 05 12:12:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Sep 05 12:15:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a907b90b-adbd-5079-3c3c-bce1be02680d (at 10.9.107.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880834346000, cur 1536174959 expire 1536174809 last 1536174732 Sep 05 12:21:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 05 12:33:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client daa168aa-4495-f309-b659-cd75a106dc84 (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88035e2e5800, cur 1536175997 expire 1536175847 last 1536175770 Sep 05 12:36:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d342f90e-597c-3ec6-8e55-42583e98e20d (at 10.9.107.31@o2ib4) Sep 05 12:39:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e5b8616-af3d-3643-e7f8-e45e674fa9b3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88046864cc00, cur 1536176379 expire 1536176229 last 1536176152 Sep 05 12:43:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 05 12:51:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 05 13:26:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0608926-7aa8-8a53-2081-8f72d582a5d7 (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102cfdb000, cur 1536179198 expire 1536179048 last 1536178971 Sep 05 13:36:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fc4b5271-be0d-0e0a-a2d7-5e5e4e3f2320 (at 10.8.14.2@o2ib6) Sep 05 13:57:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e8f82620-e480-b4c7-f7a4-247bcbefc4a2 (at 10.8.0.68@o2ib6) Sep 05 13:59:54 oak-md1-s2 kernel: LNet: 464:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.251@o2ib5 version 12/12 incarnation 1536017787804314/1536181167042077 Sep 05 13:59:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 684d53b4-3df6-9d5b-0506-1d1fcbfe8402 (at 10.0.2.251@o2ib5) Sep 05 14:04:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) Sep 05 14:13:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 66eae18a-e135-0c44-6bd6-966c4fe09ba0 (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea6800, cur 1536182008 expire 1536181858 last 1536181781 Sep 05 14:30:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fc4b5271-be0d-0e0a-a2d7-5e5e4e3f2320 (at 10.8.14.2@o2ib6) Sep 05 14:48:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42e493fa-5839-47ad-6bbd-80f855278552 (at 10.9.113.8@o2ib4) Sep 05 14:49:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 05 15:49:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aff9ef7e-cabe-7f52-fc25-fceafef993c0 (at 10.8.3.9@o2ib6) Sep 05 15:55:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c676738d-99c7-0eac-2f4d-5ff2ba09454d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8657e400, cur 1536188118 expire 1536187968 last 1536187891 Sep 05 16:04:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aff9ef7e-cabe-7f52-fc25-fceafef993c0 (at 10.8.3.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f15cb7c00, cur 1536188652 expire 1536188502 last 1536188425 Sep 05 16:05:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aff9ef7e-cabe-7f52-fc25-fceafef993c0 (at 10.8.3.9@o2ib6) Sep 05 16:06:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d7e2740e-a189-449d-6d52-524bb16acc20 (at 10.8.3.16@o2ib6) Sep 05 16:14:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 05 16:19:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 05 16:23:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 05 16:23:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 88894ac4-5676-8d42-da5f-6fffa235d4f5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807fa9ae000, cur 1536189835 expire 1536189685 last 1536189608 Sep 05 16:28:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0725839a-d273-035c-626e-5d34ad9e18a2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880530ed0c00, cur 1536190089 expire 1536189939 last 1536189862 Sep 05 16:35:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aff9ef7e-cabe-7f52-fc25-fceafef993c0 (at 10.8.3.9@o2ib6) Sep 05 17:17:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04d316c3-ab5b-e162-c5a9-07140cab1840 (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803942a5400, cur 1536193066 expire 1536192916 last 1536192839 Sep 05 17:26:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 05 18:00:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 64af459a-c751-728c-4a25-c6081eed2f3b (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102cfdd000, cur 1536195613 expire 1536195463 last 1536195386 Sep 05 18:46:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 986bf173-8a48-8326-0b87-c02bc7db3b53 (at 10.8.9.8@o2ib6) Sep 05 18:47:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 986bf173-8a48-8326-0b87-c02bc7db3b53 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801393cc000, cur 1536198474 expire 1536198324 last 1536198247 Sep 05 19:15:00 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 20:38:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4730bcf5-3e13-531a-f439-236a1688b0a3 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea0aff800, cur 1536205120 expire 1536204970 last 1536204893 Sep 05 20:57:49 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:07:57 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:08:17 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:40:51 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:50:56 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:53:18 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:53:20 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:53:20 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 44 previous similar messages Sep 05 21:53:24 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 21:53:24 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 234 previous similar messages Sep 05 22:29:32 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 22:29:32 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 277 previous similar messages Sep 05 22:36:41 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 05 23:11:58 oak-md1-s2 kernel: Lustre: 10566:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 00:22:43 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 01:00:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02403ab7-dda9-5d1c-de85-b50ffb7be273 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dc4522000, cur 1536220834 expire 1536220684 last 1536220607 Sep 06 01:23:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 06 04:04:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e6af7493-5c77-067f-6cee-37f589a3ba62 (at 10.8.2.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810392a2800, cur 1536231881 expire 1536231731 last 1536231654 Sep 06 04:28:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 403d2249-61ba-089a-a8c6-8ccf250b6567 (at 10.8.1.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a99800, cur 1536233306 expire 1536233156 last 1536233079 Sep 06 06:00:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d85dd6b-c92b-660d-fcaa-6da2163a1f77 (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880814643c00, cur 1536238831 expire 1536238681 last 1536238604 Sep 06 09:00:27 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 09:00:27 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 06 09:02:16 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 09:16:08 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 09:16:08 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 06 09:16:09 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 09:16:09 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 06 09:28:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 506c0f13-dc3c-6a73-0a81-4ba5be015438 (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e38759000, cur 1536251313 expire 1536251163 last 1536251086 Sep 06 09:35:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0b334521-e8aa-58c6-0fd8-449db0274f2c (at 10.8.17.30@o2ib6) Sep 06 09:36:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79664972-5809-1d71-1f47-39d431d7fe8b (at 10.8.17.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807fe5a7800, cur 1536251785 expire 1536251635 last 1536251558 Sep 06 09:39:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0b334521-e8aa-58c6-0fd8-449db0274f2c (at 10.8.17.30@o2ib6) Sep 06 09:40:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5f9e104-52b4-3fb8-30ba-787464c0c14f (at 10.8.17.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102dfee000, cur 1536252042 expire 1536251892 last 1536251815 Sep 06 09:45:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0b334521-e8aa-58c6-0fd8-449db0274f2c (at 10.8.17.30@o2ib6) Sep 06 09:46:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30646684-5930-24f8-4745-eea73dcc53c1 (at 10.8.17.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f9d204000, cur 1536252383 expire 1536252233 last 1536252156 Sep 06 09:49:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0b51fbe-4478-6b3a-9645-4c65443b1eb7 (at 10.8.17.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809409c7c00, cur 1536252569 expire 1536252419 last 1536252342 Sep 06 10:07:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Sep 06 10:26:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) Sep 06 10:27:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d85dd6b-c92b-660d-fcaa-6da2163a1f77 (at 10.8.1.14@o2ib6) Sep 06 10:27:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e6af7493-5c77-067f-6cee-37f589a3ba62 (at 10.8.2.19@o2ib6) Sep 06 10:51:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c9a55d2c-13eb-8393-c825-58d5a0d71edf (at 10.8.6.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be64800, cur 1536256298 expire 1536256148 last 1536256071 Sep 06 10:54:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5205f4ef-dc22-9777-bed7-c7285041ba31 (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe94e0c00, cur 1536256460 expire 1536256310 last 1536256233 Sep 06 11:06:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c9a55d2c-13eb-8393-c825-58d5a0d71edf (at 10.8.6.16@o2ib6) Sep 06 11:12:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Sep 06 12:19:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 06 12:22:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6094ad6e-8dae-6baa-a23e-ab1f7c316cac (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880150efcc00, cur 1536261730 expire 1536261580 last 1536261503 Sep 06 14:25:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5a816ffb-848a-a9f8-e400-c8e354652630 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8699c00, cur 1536269101 expire 1536268951 last 1536268874 Sep 06 14:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client acb78d5c-ce52-9a44-b71b-f19d6c82b7dc (at 10.8.16.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8820098d9400, cur 1536270825 expire 1536270675 last 1536270598 Sep 06 15:06:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acb78d5c-ce52-9a44-b71b-f19d6c82b7dc (at 10.8.16.6@o2ib6) Sep 06 16:36:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880150efc000, cur 1536277018 expire 1536276868 last 1536276791 Sep 06 17:01:41 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 17:34:55 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 17:34:55 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 06 18:02:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 70a62f37-33fa-c309-cda0-a634ba75bf14 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9f000, cur 1536282157 expire 1536282007 last 1536281930 Sep 06 18:04:52 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 18:44:22 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 18:45:56 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 18:57:21 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 18:57:21 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 06 19:01:29 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 19:01:29 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 06 19:01:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 618788cf-c753-b0e1-5d78-7ff223b8054e (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096a566400, cur 1536285715 expire 1536285565 last 1536285488 Sep 06 19:09:57 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 19:10:06 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 19:10:06 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Sep 06 19:11:14 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 19:11:14 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Sep 06 19:11:22 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 19:11:22 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 40 previous similar messages Sep 06 19:12:32 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 19:12:32 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 85 previous similar messages Sep 06 19:15:17 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 06 19:15:17 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 61 previous similar messages Sep 07 00:16:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ff77b8d-b85d-6077-7091-ce9b634eda8d (at 10.8.2.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880963366000, cur 1536304597 expire 1536304447 last 1536304370 Sep 07 00:27:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8ff77b8d-b85d-6077-7091-ce9b634eda8d (at 10.8.2.7@o2ib6) Sep 07 00:49:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 07 00:51:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2019abe4-ae70-e7f0-d0e2-02784e88f322 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e902b8400, cur 1536306673 expire 1536306523 last 1536306446 Sep 07 07:13:39 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 07 07:13:39 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 07 07:31:29 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 07 08:35:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a816ffb-848a-a9f8-e400-c8e354652630 (at 10.8.14.7@o2ib6) Sep 07 08:51:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a816ffb-848a-a9f8-e400-c8e354652630 (at 10.8.14.7@o2ib6) Sep 07 08:57:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 07 08:57:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 07 09:40:35 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 07 09:50:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8271ebc4-d739-add6-952f-6177412af161 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f63e800, cur 1536339025 expire 1536338875 last 1536338798 Sep 07 11:33:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 07 11:35:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8271ebc4-d739-add6-952f-6177412af161 (at 10.9.112.16@o2ib4) Sep 07 12:24:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 13:07:37 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 07 14:05:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f3d77d89-daf3-1955-f228-863b5222cafc (at 10.12.4.63@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be03800, cur 1536354325 expire 1536354175 last 1536354098 Sep 07 14:52:25 oak-md1-s2 kernel: Lustre: 10639:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1536357144/real 1536357145] req@ffff8803ad958c00 x1610626761179152/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1536357151 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Sep 07 14:52:32 oak-md1-s2 kernel: Lustre: 10639:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536357145/real 1536357145] req@ffff8803ad958c00 x1610626761179152/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1536357152 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 07 14:52:39 oak-md1-s2 kernel: Lustre: 10639:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536357152/real 1536357152] req@ffff8803ad958c00 x1610626761179152/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1536357159 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 07 14:52:46 oak-md1-s2 kernel: Lustre: 10639:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536357159/real 1536357159] req@ffff8803ad958c00 x1610626761179152/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1536357166 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 07 14:52:54 oak-md1-s2 kernel: Lustre: 10639:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536357166/real 1536357166] req@ffff8803ad958c00 x1610626761179152/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1536357173 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 07 14:53:08 oak-md1-s2 kernel: Lustre: 10639:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536357181/real 1536357181] req@ffff8803ad958c00 x1610626761179152/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1536357188 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 07 14:53:08 oak-md1-s2 kernel: Lustre: 10639:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 07 14:53:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 357c9327-5943-7b6b-bf71-96f316a93e47 (at 10.0.2.247@o2ib5) Sep 07 14:53:22 oak-md1-s2 kernel: LustreError: 10639:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.0.2.247@o2ib5) returned error from blocking AST (req@ffff8803ad958c00 x1610626761179152 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880503bd6c00/0x6c00c3d80256d657 lrc: 4/0,0 mode: PR/PR res: [0x2000013b2:0xe:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.0.2.247@o2ib5 remote: 0x27034f61106d925c expref: 456744 pid: 10633 timeout: 4642295063 lvb_type: 0 Sep 07 14:53:22 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.0.2.247@o2ib5 was evicted due to a lock blocking callback time out: rc -107 Sep 07 17:45:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1db58f68-7c4f-2629-6fce-82554b89fc05 (at 10.8.6.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e72c52800, cur 1536367551 expire 1536367401 last 1536367324 Sep 07 18:22:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1db58f68-7c4f-2629-6fce-82554b89fc05 (at 10.8.6.9@o2ib6) Sep 07 19:58:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 19:58:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ad29747c-1e00-2f5e-ffaf-589c63ef5e2e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880adc271800, cur 1536375528 expire 1536375378 last 1536375301 Sep 07 20:20:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 986bf173-8a48-8326-0b87-c02bc7db3b53 (at 10.8.9.8@o2ib6) Sep 07 20:22:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d791b063-5c56-2016-007b-e306768489c5 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b28fda800, cur 1536376933 expire 1536376783 last 1536376706 Sep 07 21:37:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 21:38:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f410946d-3c9d-a22c-06bd-dccf234b0319 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e61cfcc00, cur 1536381484 expire 1536381334 last 1536381257 Sep 07 21:43:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7e9d8c0c-ad44-a85a-2d48-4c0c297ec9ca (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc4f61c00, cur 1536381837 expire 1536381687 last 1536381610 Sep 07 21:59:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 22:03:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a6be150-7112-d126-d00f-b41d7311110f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e19c46400, cur 1536383005 expire 1536382855 last 1536382778 Sep 07 22:12:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 22:17:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 22:18:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a65effa3-c153-2ade-9175-d288b77f2c13 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf200400, cur 1536383882 expire 1536383732 last 1536383655 Sep 07 22:23:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 22:23:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 77e7dcae-6dba-c924-80d9-a3d762a332af (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c46166000, cur 1536384223 expire 1536384073 last 1536383996 Sep 07 22:27:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 22:28:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5269df0e-4fd9-0830-3791-8938c2682d4c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001db41800, cur 1536384494 expire 1536384344 last 1536384267 Sep 07 22:32:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe51d6cf-3cc8-30a2-72f8-cc5fdbd5fd2f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e17d99c00, cur 1536384778 expire 1536384628 last 1536384551 Sep 07 22:42:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 07 22:47:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d7fbb82-d245-d825-43a1-b3b27011740c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe9400800, cur 1536385631 expire 1536385481 last 1536385404 Sep 07 22:56:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:05:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:05:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f3a93aa-6faf-24ba-3f4d-0aafd7966cbb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf074000, cur 1536390328 expire 1536390178 last 1536390101 Sep 08 00:10:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:10:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9be52bb-a591-bcee-f1a2-1feed4f784dd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa18fb400, cur 1536390630 expire 1536390480 last 1536390403 Sep 08 00:15:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 18ca48fb-91cd-83e3-3d6d-4b9f48adaaf6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803671d5c00, cur 1536390925 expire 1536390775 last 1536390698 Sep 08 00:16:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:20:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:20:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 409532eb-c7d8-ace5-9fc5-698e6f36fe83 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803586a2800, cur 1536391234 expire 1536391084 last 1536391007 Sep 08 00:24:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:25:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a31adf43-79c0-d2be-fe11-08c1580206b5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881beeebf800, cur 1536391506 expire 1536391356 last 1536391279 Sep 08 00:29:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:30:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 418314ac-52e0-f108-b1a9-3ac0c7d84e99 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88079195ec00, cur 1536391807 expire 1536391657 last 1536391580 Sep 08 00:35:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0921c355-25f3-5f01-e053-3a24edf8cfe8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806a1239000, cur 1536392128 expire 1536391978 last 1536391901 Sep 08 00:44:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 00:51:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7198607d-de7f-a2c9-7ff8-c5ba8eccc29c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bb53ff800, cur 1536393109 expire 1536392959 last 1536392882 Sep 08 01:00:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 01:08:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bdd4a44-dada-916a-d48e-32764ae43da1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881891f34c00, cur 1536394104 expire 1536393954 last 1536393877 Sep 08 01:17:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 01:24:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 01:24:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 34d58090-7d4a-4650-f348-d2c234eec447 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e8d83ac00, cur 1536395074 expire 1536394924 last 1536394847 Sep 08 01:30:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 01:30:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe8dcb30-20d5-f34e-6db2-485a21de9d4c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9f400, cur 1536395453 expire 1536395303 last 1536395226 Sep 08 01:36:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 01:36:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4a235e6-2a7f-f6f9-de96-ab78e3a583c8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804c2ece000, cur 1536395813 expire 1536395663 last 1536395586 Sep 08 01:42:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a5c61703-639f-5b76-7f7b-c85e26b651c1 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a978fb000, cur 1536396129 expire 1536395979 last 1536395902 Sep 08 01:48:19 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 02:28:07 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 03:07:15 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 03:07:16 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 03:07:16 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Sep 08 03:07:17 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 03:07:17 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 43 previous similar messages Sep 08 03:07:19 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 03:07:19 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 105 previous similar messages Sep 08 03:07:23 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 03:07:23 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 208 previous similar messages Sep 08 03:15:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 811d9648-111d-4b9d-4d6c-296ea56b239e (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ae9d8c000, cur 1536401756 expire 1536401606 last 1536401529 Sep 08 04:08:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f97f5c4e-ea1b-a56f-2caf-9e8a029f9f0e (at 10.12.4.30@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039010c00, cur 1536404909 expire 1536404759 last 1536404682 Sep 08 04:15:09 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 04:15:09 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 298 previous similar messages Sep 08 05:47:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5bf085c4-7698-cb73-d91c-d388aeb8271a (at 10.8.2.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881030653c00, cur 1536410850 expire 1536410700 last 1536410623 Sep 08 05:51:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5bf085c4-7698-cb73-d91c-d388aeb8271a (at 10.8.2.14@o2ib6) Sep 08 09:26:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 09:27:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 452f081d-5f7e-ddfb-f18f-ef92d3d20b94 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bf4dae400, cur 1536424037 expire 1536423887 last 1536423810 Sep 08 09:32:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a26adc3-07da-3e6c-2c5f-84b60fdf5df3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca2000, cur 1536424345 expire 1536424195 last 1536424118 Sep 08 09:41:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 11:22:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 11:22:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04131ac2-60b8-ddc9-2705-38d745bcf6e6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88075eba6c00, cur 1536430971 expire 1536430821 last 1536430744 Sep 08 12:58:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e99eca34-37bb-bfb5-e26d-1e0a99cf331e (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb89000, cur 1536436685 expire 1536436535 last 1536436458 Sep 08 14:39:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30273261-9a46-c7c6-cbad-38289f3da03e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041f137800, cur 1536442795 expire 1536442645 last 1536442568 Sep 08 14:48:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 14:53:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 14:53:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 89e07176-8ed1-2051-75c1-e26149744a79 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0e1e0c00, cur 1536443607 expire 1536443457 last 1536443380 Sep 08 14:56:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 14:57:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa563c40-4739-0030-9355-99430d15b427 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c6f6edc00, cur 1536443820 expire 1536443670 last 1536443593 Sep 08 15:00:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 15:00:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 337b41b3-6e52-a1d4-79e6-f65253ecdedd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88035a577400, cur 1536444031 expire 1536443881 last 1536443804 Sep 08 16:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881038287000, cur 1536447720 expire 1536447570 last 1536447493 Sep 08 19:02:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6fbfd0d4-39d0-9d8c-60fd-5e52ae34723a (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880541155400, cur 1536458524 expire 1536458374 last 1536458297 Sep 08 19:46:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e02a5e09-3091-c87d-d666-026750b73bcf (at 10.8.6.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880463a37400, cur 1536461182 expire 1536461032 last 1536460955 Sep 08 19:51:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c9a55d2c-13eb-8393-c825-58d5a0d71edf (at 10.8.6.16@o2ib6) Sep 08 21:03:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 789d7a4b-e2ea-b38c-c6a7-e8527d4924d1 (at 10.8.6.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c26c78800, cur 1536465835 expire 1536465685 last 1536465608 Sep 08 22:13:36 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 08 22:23:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d1328d7-1f65-43ec-993b-641daa2d8350 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bde5ae000, cur 1536470594 expire 1536470444 last 1536470367 Sep 08 22:28:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 22:28:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f97f5c4e-ea1b-a56f-2caf-9e8a029f9f0e (at 10.12.4.30@o2ib) Sep 08 22:32:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 22:33:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc56e44b-bbb6-e4de-6630-ad9fb2e8436e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880954f8c800, cur 1536471188 expire 1536471038 last 1536470961 Sep 08 22:36:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e64c2164-25e1-8f23-f482-928fb8833662 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cf766a000, cur 1536471373 expire 1536471223 last 1536471146 Sep 08 22:46:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 22:50:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 22:51:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e57a401-2eed-36d3-6569-182ccc7617e5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aedf7b400, cur 1536472260 expire 1536472110 last 1536472033 Sep 08 22:54:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 22:54:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e240c00-4e87-368c-b497-43b3a9111e43 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805fbfa7000, cur 1536472476 expire 1536472326 last 1536472249 Sep 08 22:58:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 22:58:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 818a4fb1-df90-f01c-c7d7-6b3502f7e779 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e8246c00, cur 1536472688 expire 1536472538 last 1536472461 Sep 08 23:03:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ba3facbf-ecc9-4a0c-18c6-00d0d0015ae6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aedf78c00, cur 1536472986 expire 1536472836 last 1536472759 Sep 08 23:03:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 23:08:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 23:08:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 947a8a8c-735f-6793-91fa-3c728897a5b8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880157626800, cur 1536473315 expire 1536473165 last 1536473088 Sep 08 23:13:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 23:13:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9884d02-aa89-b452-5bf5-54319e86b725 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f03e1d800, cur 1536473612 expire 1536473462 last 1536473385 Sep 08 23:18:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb0877e9-6a1c-1cd8-3dbe-a7cc72c9620c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f03e1c000, cur 1536473887 expire 1536473737 last 1536473660 Sep 08 23:27:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 23:40:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 08 23:40:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fc74782f-33a7-e76b-163e-fc8bcd176b6f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f15cb3000, cur 1536475233 expire 1536475083 last 1536475006 Sep 09 00:44:15 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 09 00:58:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e17d9f000, cur 1536479893 expire 1536479743 last 1536479666 Sep 09 02:16:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dda81b71-06cf-ea94-5026-8410514a7ac3 (at 10.12.4.72@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be05800, cur 1536484600 expire 1536484450 last 1536484373 Sep 09 04:29:21 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 09 07:04:01 oak-md1-s2 kernel: Lustre: 3869:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 09 09:46:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103a05e400, cur 1536511598 expire 1536511448 last 1536511371 Sep 09 09:47:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Sep 09 09:49:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53462e99-45fa-ed22-9be8-74feb94dacbf (at 10.8.12.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a45b71c00, cur 1536511744 expire 1536511594 last 1536511517 Sep 09 09:59:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 53462e99-45fa-ed22-9be8-74feb94dacbf (at 10.8.12.6@o2ib6) Sep 09 15:19:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 09 15:21:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9d002ee4-45cf-cc39-8442-cb62cae06702 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ee806c00, cur 1536531696 expire 1536531546 last 1536531469 Sep 09 17:25:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 09 17:27:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e17d99000, cur 1536539230 expire 1536539080 last 1536539003 Sep 09 17:33:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bfd066a8-e3b7-5f5d-fa85-eff2a3f63637 (at 10.8.14.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880950a8d400, cur 1536539632 expire 1536539482 last 1536539405 Sep 09 19:35:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 32fe5935-c3dc-95ed-1ed7-3351a730468d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806a9e65c00, cur 1536546923 expire 1536546773 last 1536546696 Sep 09 19:40:04 oak-md1-s2 kernel: Lustre: 10508:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 09 19:44:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 09 19:49:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 09 19:50:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 62b28b24-24c6-ae6b-5eac-4da8a74f4f2e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4758c800, cur 1536547810 expire 1536547660 last 1536547583 Sep 09 19:53:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 09 19:54:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ecec2f9f-5bf1-9aab-e761-51bf20313b31 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080dbc2000, cur 1536548087 expire 1536547937 last 1536547860 Sep 09 19:55:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to edf32d0b-88eb-f1d4-00b8-c0af2b7893e4 (at 10.12.4.38@o2ib) Sep 09 19:58:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 09 19:58:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aa3f1135-71d8-cb89-1dae-1545aa50ecc2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93137400, cur 1536548329 expire 1536548179 last 1536548102 Sep 09 19:58:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 09 20:43:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97a5006d-4df9-85a1-2010-81dbe0192ab3 (at 10.9.106.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092ea03c00, cur 1536550986 expire 1536550836 last 1536550759 Sep 09 22:22:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882005c36c00, cur 1536556940 expire 1536556790 last 1536556713 Sep 09 22:22:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 09 22:23:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Sep 10 00:23:01 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 00:23:02 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 04:50:31 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 05:14:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e17d9bc00, cur 1536581693 expire 1536581543 last 1536581466 Sep 10 05:14:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 10 06:24:57 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 06:24:58 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 08:19:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 10 08:20:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Sep 10 08:22:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 96afcbea-9310-e67f-1fe7-5fae0374f13b (at 10.9.106.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a106a2000, cur 1536592950 expire 1536592800 last 1536592723 Sep 10 08:25:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8271ebc4-d739-add6-952f-6177412af161 (at 10.9.112.16@o2ib4) Sep 10 08:41:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 10 08:42:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9effb4e8-f195-a3eb-e68a-859b1f5dda95 (at 10.8.14.4@o2ib6) Sep 10 08:48:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 97a5006d-4df9-85a1-2010-81dbe0192ab3 (at 10.9.106.67@o2ib4) Sep 10 09:05:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5be35293-3690-ba71-55ac-1eda538771b5 (at 10.8.2.3@o2ib6) Sep 10 09:06:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 10 09:07:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 10 09:10:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7e924def-a27e-9720-a88d-67894893f2c5 (at 10.8.2.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88020cea2400, cur 1536595817 expire 1536595667 last 1536595590 Sep 10 09:14:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 10 09:25:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to efd35555-5fc0-7307-e511-6d9f56f7bb12 (at 10.8.3.5@o2ib6) Sep 10 09:33:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to feeee0a9-94d1-55ba-1a7d-affde36dfe4f (at 10.8.2.4@o2ib6) Sep 10 09:34:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9be6edfa-ebde-1d5d-f312-2249d8fda40b (at 10.8.2.11@o2ib6) Sep 10 09:34:12 oak-md1-s2 kernel: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z) Sep 10 09:34:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1585ed96-ba3a-af1b-3b88-056c0a1fcdb8 (at 10.8.2.2@o2ib6) Sep 10 09:42:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5be35293-3690-ba71-55ac-1eda538771b5 (at 10.8.2.3@o2ib6) Sep 10 09:45:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2012ec85-a6fc-f7ce-e763-43d3a322b64e (at 10.8.3.14@o2ib6) Sep 10 09:46:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34974ccb-c1b1-bcb8-835a-466ad18e87c2 (at 10.8.3.20@o2ib6) Sep 10 09:46:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 10 09:52:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f3d77d89-daf3-1955-f228-863b5222cafc (at 10.12.4.63@o2ib) Sep 10 09:56:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b4760c16-3ca5-38e2-8cc1-a208adb3cabf (at 10.8.3.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806a9e63000, cur 1536598598 expire 1536598448 last 1536598371 Sep 10 10:20:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 64709e3d-aec3-8528-aefd-ad77d000a881 (at 10.8.3.30@o2ib6) Sep 10 10:20:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2012ec85-a6fc-f7ce-e763-43d3a322b64e (at 10.8.3.14@o2ib6) Sep 10 10:20:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 10 10:20:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b766b845-24a8-fe19-d883-9cb881fb19ed (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88072bf7f800, cur 1536600032 expire 1536599882 last 1536599805 Sep 10 10:20:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 10 10:21:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59705672-969a-abd1-3334-80089993f865 (at 10.8.4.4@o2ib6) in 154 seconds. I think it's dead, and I am evicting it. exp ffff880954f8b400, cur 1536600108 expire 1536599958 last 1536599954 Sep 10 10:31:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 51435d4a-c2fd-a2da-4ac9-32a53915c2d0 (at 10.8.3.33@o2ib6) Sep 10 10:31:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34974ccb-c1b1-bcb8-835a-466ad18e87c2 (at 10.8.3.20@o2ib6) Sep 10 10:45:33 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:03:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3a815cbc-d650-5201-c8ac-6586d80e6809 (at 10.9.101.41@o2ib4) Sep 10 11:04:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2eb5c032-e495-d959-c1b3-624fdefab331 (at 10.8.4.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a76abd400, cur 1536602650 expire 1536602500 last 1536602423 Sep 10 11:05:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2eb5c032-e495-d959-c1b3-624fdefab331 (at 10.8.4.33@o2ib6) Sep 10 11:15:18 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:19:41 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:19:55 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:19:55 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 10 11:20:00 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:20:00 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Sep 10 11:20:15 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:20:15 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages Sep 10 11:20:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43a05b34-4ef7-6ca4-916f-2782eebf9a37 (at 10.8.4.26@o2ib6) Sep 10 11:21:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Sep 10 11:23:25 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:23:25 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 10 11:24:18 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:24:54 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:24:54 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Sep 10 11:27:52 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:27:52 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 59 previous similar messages Sep 10 11:28:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a96ae6b2-7b2b-728e-17e0-eaf40405df4a (at 10.8.3.19@o2ib6) Sep 10 11:30:01 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:30:01 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 111 previous similar messages Sep 10 11:30:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eaee074c-c42a-1b0d-d24f-6a4e0ab844fd (at 10.8.3.10@o2ib6) Sep 10 11:34:18 oak-md1-s2 kernel: Lustre: 10555:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:34:18 oak-md1-s2 kernel: Lustre: 10555:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 998 previous similar messages Sep 10 11:35:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 825e25d6-46ba-66b5-860a-7df142eb25ba (at 10.8.4.13@o2ib6) Sep 10 11:42:51 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:42:51 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5471 previous similar messages Sep 10 11:49:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 10 11:51:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3f3dd7d-c855-233a-0702-474594ae6684 (at 10.8.4.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a76abec00, cur 1536605470 expire 1536605320 last 1536605243 Sep 10 11:51:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 10 11:53:06 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 11:53:06 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14119 previous similar messages Sep 10 11:53:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Sep 10 11:58:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9638f799-7c88-d188-f651-da3d16004a31 (at 10.8.4.34@o2ib6) Sep 10 12:03:11 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 12:03:11 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12114 previous similar messages Sep 10 12:06:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48ec7f74-5b7c-f0ad-fb35-6041a89f83d3 (at 10.8.4.16@o2ib6) Sep 10 12:13:13 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 12:13:13 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13184 previous similar messages Sep 10 12:23:18 oak-md1-s2 kernel: Lustre: 3869:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 12:23:18 oak-md1-s2 kernel: Lustre: 3869:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17392 previous similar messages Sep 10 12:31:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) reconnecting Sep 10 12:31:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) Sep 10 12:33:22 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 12:33:22 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16227 previous similar messages Sep 10 12:43:24 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 12:43:24 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14980 previous similar messages Sep 10 12:53:26 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 12:53:26 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10530 previous similar messages Sep 10 12:53:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6cdfa4a1-0eb5-fc32-84dc-5732f06d5d85 (at 10.8.4.7@o2ib6) Sep 10 12:57:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 96afcbea-9310-e67f-1fe7-5fae0374f13b (at 10.9.106.25@o2ib4) Sep 10 12:59:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3f3dd7d-c855-233a-0702-474594ae6684 (at 10.8.4.24@o2ib6) Sep 10 13:03:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 62d62ce3-c899-dd75-eb18-cc46060a0904 (at 10.8.4.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d5f6000, cur 1536609789 expire 1536609639 last 1536609562 Sep 10 13:03:09 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 10 13:03:29 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 13:03:29 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18775 previous similar messages Sep 10 13:09:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97dfe01e-35f4-5412-8ea1-fa2cbaeb7df9 (at 10.8.4.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880002bef400, cur 1536610186 expire 1536610036 last 1536609959 Sep 10 13:13:32 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 13:13:32 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21377 previous similar messages Sep 10 13:20:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Sep 10 13:23:33 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 13:23:33 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14949 previous similar messages Sep 10 13:30:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9411437c-225a-bf4a-e991-7fd9cb0edb7a (at 10.8.6.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802c8b62400, cur 1536611409 expire 1536611259 last 1536611182 Sep 10 13:30:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 10 13:33:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b067ccdd-27cb-ae5c-27d9-df62cb606375 (at 10.8.4.19@o2ib6) Sep 10 13:33:46 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 13:33:46 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14665 previous similar messages Sep 10 13:37:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9638f799-7c88-d188-f651-da3d16004a31 (at 10.8.4.34@o2ib6) Sep 10 13:43:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49436388-6e40-0ef9-1610-3273dbbd6cd8 (at 10.8.4.10@o2ib6) Sep 10 13:43:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6cdfa4a1-0eb5-fc32-84dc-5732f06d5d85 (at 10.8.4.7@o2ib6) Sep 10 13:44:01 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 13:44:01 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13113 previous similar messages Sep 10 13:44:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3f3dd7d-c855-233a-0702-474594ae6684 (at 10.8.4.24@o2ib6) Sep 10 13:50:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 10 13:51:41 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536612694/real 1536612694] req@ffff880ed12db300 x1610626924729808/t0(0) o104->oak-MDT0000@10.8.15.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536612701 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 10 13:51:41 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 10 13:51:49 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536612701/real 1536612701] req@ffff880ed12db300 x1610626924729808/t0(0) o104->oak-MDT0000@10.8.15.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536612708 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 10 13:51:56 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536612709/real 1536612709] req@ffff880ed12db300 x1610626924729808/t0(0) o104->oak-MDT0000@10.8.15.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536612716 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 10 13:52:10 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536612723/real 1536612723] req@ffff880ed12db300 x1610626924729808/t0(0) o104->oak-MDT0000@10.8.15.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536612730 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 10 13:52:10 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 10 13:52:31 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536612744/real 1536612744] req@ffff880ed12db300 x1610626924729808/t0(0) o104->oak-MDT0000@10.8.15.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536612751 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 10 13:52:31 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 10 13:53:06 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536612779/real 1536612779] req@ffff880ed12db300 x1610626924729808/t0(0) o104->oak-MDT0000@10.8.15.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536612786 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 10 13:53:06 oak-md1-s2 kernel: Lustre: 10564:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 10 13:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e95cc81b-d945-25d9-6f2f-d1aff692d1eb (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880860fa2c00, cur 1536612825 expire 1536612675 last 1536612598 Sep 10 13:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 189897b2-be25-1d7e-aa38-95ade6914239 (at 10.8.4.31@o2ib6) Sep 10 13:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Export ffff880067359000 already connecting from 10.8.4.31@o2ib6 Sep 10 13:54:02 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 13:54:02 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8222 previous similar messages Sep 10 13:54:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 189897b2-be25-1d7e-aa38-95ade6914239 (at 10.8.4.31@o2ib6) reconnecting Sep 10 13:54:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 189897b2-be25-1d7e-aa38-95ade6914239 (at 10.8.4.31@o2ib6) Sep 10 14:01:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3796746c-13cb-1dff-031b-f34216a2f802 (at 10.8.4.18@o2ib6) Sep 10 14:01:35 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 10 14:04:02 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 14:04:02 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13749 previous similar messages Sep 10 14:04:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 455e8050-19a4-2e12-d889-3d939ef346af (at 10.8.6.24@o2ib6) Sep 10 14:04:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 755d9e41-0c85-1a53-56eb-c8b66d3c94fd (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210cc00, cur 1536613474 expire 1536613324 last 1536613247 Sep 10 14:14:07 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 14:14:07 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12162 previous similar messages Sep 10 14:24:07 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 14:24:07 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14405 previous similar messages Sep 10 14:29:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ae1f97b-3b8a-2ade-720b-e285c80acdc5 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880468649c00, cur 1536614959 expire 1536614809 last 1536614732 Sep 10 14:34:16 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 14:34:16 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21094 previous similar messages Sep 10 14:40:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) reconnecting Sep 10 14:40:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) Sep 10 14:44:16 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 14:44:16 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16844 previous similar messages Sep 10 14:54:17 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 14:54:17 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16524 previous similar messages Sep 10 15:04:29 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 15:04:29 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16452 previous similar messages Sep 10 15:14:34 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 15:14:34 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18906 previous similar messages Sep 10 15:24:35 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 15:24:35 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10183 previous similar messages Sep 10 15:34:49 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 15:34:49 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19529 previous similar messages Sep 10 15:44:51 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 15:44:51 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14251 previous similar messages Sep 10 15:47:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) reconnecting Sep 10 15:47:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) Sep 10 15:54:58 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 15:54:58 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22136 previous similar messages Sep 10 15:59:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) reconnecting Sep 10 15:59:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) Sep 10 16:05:14 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 16:05:14 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 33792 previous similar messages Sep 10 16:11:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880970fb9c00, cur 1536621065 expire 1536620915 last 1536620838 Sep 10 16:15:15 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 16:15:15 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 25052 previous similar messages Sep 10 16:21:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 10 16:25:22 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 16:25:22 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21145 previous similar messages Sep 10 16:35:24 oak-md1-s2 kernel: Lustre: 10530:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 16:35:24 oak-md1-s2 kernel: Lustre: 10530:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16210 previous similar messages Sep 10 16:45:35 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 16:45:35 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18819 previous similar messages Sep 10 16:55:35 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 16:55:35 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 28521 previous similar messages Sep 10 16:57:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a83745ec-b647-ef76-7a73-4b115eaaea90 (at 10.8.2.20@o2ib6) reconnecting Sep 10 16:57:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a83745ec-b647-ef76-7a73-4b115eaaea90 (at 10.8.2.20@o2ib6) Sep 10 17:05:36 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 17:05:36 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18666 previous similar messages Sep 10 17:15:36 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 17:15:36 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18541 previous similar messages Sep 10 17:17:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6021fa6b-2139-ebb4-b81e-00f7c71d0a02 (at 10.8.6.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800729ab400, cur 1536625041 expire 1536624891 last 1536624814 Sep 10 17:22:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 84e38c95-92c2-d0e1-c9fd-b8da601b534f (at 10.8.6.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053b3fc00, cur 1536625344 expire 1536625194 last 1536625117 Sep 10 17:22:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 10 17:22:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dda81b71-06cf-ea94-5026-8410514a7ac3 (at 10.12.4.72@o2ib) Sep 10 17:25:44 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 17:25:44 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17988 previous similar messages Sep 10 17:35:47 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 17:35:47 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13275 previous similar messages Sep 10 17:45:50 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 17:45:50 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15146 previous similar messages Sep 10 17:46:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bacf9e96-cd21-585b-d541-d7c07c2a4eaa (at 10.8.6.33@o2ib6) Sep 10 17:46:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 56c644c5-922f-0c6b-a7ec-8f493c7e6a52 (at 10.8.6.25@o2ib6) Sep 10 17:46:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 455e8050-19a4-2e12-d889-3d939ef346af (at 10.8.6.24@o2ib6) Sep 10 17:46:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b15183a6-3a40-f2ab-391b-493af7b7ad4f (at 10.8.6.36@o2ib6) Sep 10 17:55:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9411437c-225a-bf4a-e991-7fd9cb0edb7a (at 10.8.6.31@o2ib6) Sep 10 17:55:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49764102-6fe6-f7c6-5244-7306703d8948 (at 10.8.6.35@o2ib6) Sep 10 17:55:51 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 17:55:51 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14383 previous similar messages Sep 10 17:57:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client acba7789-d9e9-a95c-2ae6-ee748f2e059f (at 10.8.6.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e3a400, cur 1536627463 expire 1536627313 last 1536627236 Sep 10 17:59:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e5dba0a1-6752-d1db-8cce-13d7bdc15bd1 (at 10.8.6.28@o2ib6) Sep 10 18:05:55 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 18:05:55 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9414 previous similar messages Sep 10 18:09:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a229ada3-99e5-45f8-0ba6-7ede88791148 (at 10.8.4.2@o2ib6) Sep 10 18:15:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a229ada3-99e5-45f8-0ba6-7ede88791148 (at 10.8.4.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f6c00, cur 1536628514 expire 1536628364 last 1536628287 Sep 10 18:15:55 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 18:15:55 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13419 previous similar messages Sep 10 18:25:56 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 18:25:56 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7749 previous similar messages Sep 10 18:27:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 096d7c4d-4f27-b76a-aa88-24fe1294a901 (at 10.8.9.2@o2ib6) Sep 10 18:29:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 096d7c4d-4f27-b76a-aa88-24fe1294a901 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe4442400, cur 1536629358 expire 1536629208 last 1536629131 Sep 10 18:30:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 10 18:35:58 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 18:35:58 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12019 previous similar messages Sep 10 18:44:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cdcd05c3-54f5-8748-3ca6-a470bb0610fb (at 10.8.4.9@o2ib6) Sep 10 18:46:03 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 18:46:03 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9062 previous similar messages Sep 10 18:56:06 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 18:56:06 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11139 previous similar messages Sep 10 19:03:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 10 19:06:11 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 19:06:11 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9619 previous similar messages Sep 10 19:16:16 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 19:16:16 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7124 previous similar messages Sep 10 19:17:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c26648fa-7ebe-2fb6-6200-13f31b24559d (at 10.8.6.30@o2ib6) Sep 10 19:22:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38446191-d718-9765-fb81-2e842da51ed8 (at 10.8.6.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003760c800, cur 1536632521 expire 1536632371 last 1536632294 Sep 10 19:23:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Sep 10 19:26:19 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 19:26:19 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6771 previous similar messages Sep 10 19:36:21 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 19:36:21 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6924 previous similar messages Sep 10 19:46:23 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 19:46:23 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6054 previous similar messages Sep 10 19:56:26 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 19:56:26 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9012 previous similar messages Sep 10 20:06:48 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 20:06:48 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7043 previous similar messages Sep 10 20:10:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd363b0b-63b5-0c4b-80f8-dab1594fdba1 (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800496b8400, cur 1536635448 expire 1536635298 last 1536635221 Sep 10 20:17:06 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 20:17:06 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6050 previous similar messages Sep 10 20:27:14 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 20:27:14 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4778 previous similar messages Sep 10 20:37:19 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 20:37:19 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4705 previous similar messages Sep 10 20:47:19 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 20:47:19 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2938 previous similar messages Sep 10 20:57:24 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 20:57:24 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2358 previous similar messages Sep 10 21:01:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 897829f1-def7-386a-b839-29d39b9824a9 (at 10.8.6.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880056998400, cur 1536638468 expire 1536638318 last 1536638241 Sep 10 21:02:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 10 21:07:31 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 21:07:31 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 291 previous similar messages Sep 10 21:17:36 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 21:17:36 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 84 previous similar messages Sep 10 21:27:40 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 21:27:40 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 299 previous similar messages Sep 10 21:38:02 oak-md1-s2 kernel: Lustre: 10566:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 21:38:02 oak-md1-s2 kernel: Lustre: 10566:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 143 previous similar messages Sep 10 21:48:41 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 21:48:41 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 25 previous similar messages Sep 10 21:59:11 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 21:59:11 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 100 previous similar messages Sep 10 22:10:55 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 22:10:55 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 71 previous similar messages Sep 10 22:16:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fca47c9f-40fa-d757-523c-b63568ff12f0 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e902bd400, cur 1536643013 expire 1536642863 last 1536642786 Sep 10 22:21:00 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 22:21:00 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Sep 10 22:22:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client edb01a27-f560-74b3-48f3-d2fa906c395c (at 10.8.4.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e1367400, cur 1536643334 expire 1536643184 last 1536643107 Sep 10 22:23:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7dc431a-f049-e1b5-21a8-c5ef3afa8302 (at 10.8.4.27@o2ib6) Sep 10 22:26:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1dd0d451-6fc3-8a05-bce3-81b0ce7653a7 (at 10.8.2.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e385800, cur 1536643610 expire 1536643460 last 1536643383 Sep 10 22:37:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1dd0d451-6fc3-8a05-bce3-81b0ce7653a7 (at 10.8.2.1@o2ib6) Sep 10 22:38:44 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 22:38:44 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 46 previous similar messages Sep 10 22:49:10 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 22:49:10 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 37 previous similar messages Sep 10 23:05:57 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 23:05:57 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Sep 10 23:11:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e18c502-3b30-9b11-c18e-c795759d4a28 (at 10.8.4.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880019fdc000, cur 1536646294 expire 1536646144 last 1536646067 Sep 10 23:12:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49436388-6e40-0ef9-1610-3273dbbd6cd8 (at 10.8.4.10@o2ib6) Sep 10 23:18:01 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 23:18:01 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages Sep 10 23:30:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94db2251-c3fe-9ccb-85d7-e2339fcb9644 (at 10.8.4.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880060316400, cur 1536647406 expire 1536647256 last 1536647179 Sep 10 23:36:08 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 23:36:08 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 10 23:56:43 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 10 23:56:43 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Sep 11 00:01:16 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 00:01:16 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 11 00:19:02 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 00:23:03 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 00:23:03 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 11 00:24:25 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 00:24:25 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 11 00:40:29 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 01:01:43 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 01:01:43 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 11 01:05:17 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 01:05:17 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 11 01:08:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec1c0729-44fd-18b5-5a06-7474e912e515 (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e6bbd3c00, cur 1536653301 expire 1536653151 last 1536653074 Sep 11 01:13:42 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 01:24:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 11 01:33:04 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 01:33:04 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 11 01:33:05 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 01:33:49 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 01:33:49 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 11 02:10:28 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 02:12:10 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 02:12:10 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 11 02:28:36 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 03:02:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client efdfda01-ae6e-9ef1-9705-f81e360dc2b5 (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aefe8d000, cur 1536660146 expire 1536659996 last 1536659919 Sep 11 03:43:13 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 03:43:13 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 11 04:26:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c7f1a0a7-f048-65e5-d94f-75ca1cf65612 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000454c400, cur 1536665207 expire 1536665057 last 1536664980 Sep 11 04:37:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 11 04:57:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e4aac65-f8fd-aaeb-a34b-4a8996715457 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880024406400, cur 1536667068 expire 1536666918 last 1536666841 Sep 11 05:07:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 11 06:51:32 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 07:21:14 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 08:23:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 11 08:23:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Sep 11 08:23:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 11 08:25:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 11 08:31:19 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 08:35:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d85dd6b-c92b-660d-fcaa-6da2163a1f77 (at 10.8.1.14@o2ib6) Sep 11 08:38:16 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 08:55:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 72e213c3-5090-e3e6-76d5-43e6eaba24ec (at 10.0.2.60@o2ib5) Sep 11 08:56:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f77f825-50d5-ddb0-2994-024ea262a2e9 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800246f3000, cur 1536681369 expire 1536681219 last 1536681142 Sep 11 09:18:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client adbf3465-c41e-f545-9990-12d40c33d174 (at 10.8.6.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103d67e000, cur 1536682706 expire 1536682556 last 1536682479 Sep 11 09:18:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 11 09:21:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8ee71612-e9fd-8c6e-9014-df0a8f3b5e25 (at 10.8.6.17@o2ib6) Sep 11 09:23:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0f62ae40-e58e-d7bc-6c96-d5eec9ff1569 (at 10.8.6.22@o2ib6) Sep 11 09:23:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e5dba0a1-6752-d1db-8cce-13d7bdc15bd1 (at 10.8.6.28@o2ib6) Sep 11 09:24:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3f83601e-2f19-ecd1-e8a4-e59a6070c6d5 (at 10.8.6.3@o2ib6) Sep 11 09:24:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5fb42f44-6d20-1500-290f-13e593b5bb24 (at 10.8.6.15@o2ib6) Sep 11 10:00:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6079d32a-9813-3a3c-6db6-a6dcaf9a0eb4 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041f137c00, cur 1536685203 expire 1536685053 last 1536684976 Sep 11 10:09:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 11 10:15:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to adbf3465-c41e-f545-9990-12d40c33d174 (at 10.8.6.11@o2ib6) Sep 11 10:17:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fe0ee6f3-c9d3-a009-f66c-29f578585e2d (at 10.8.6.12@o2ib6) Sep 11 10:18:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 11 10:20:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3219c29b-d643-c9cf-8fd6-2446902443f8 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aca2c5c00, cur 1536686408 expire 1536686258 last 1536686181 Sep 11 10:22:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1fa0d266-5ffe-1c41-6f98-f3e707c32206 (at 10.8.6.12@o2ib6) reconnecting Sep 11 10:22:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fe0ee6f3-c9d3-a009-f66c-29f578585e2d (at 10.8.6.12@o2ib6) Sep 11 10:24:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c26648fa-7ebe-2fb6-6200-13f31b24559d (at 10.8.6.30@o2ib6) Sep 11 10:26:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8ee71612-e9fd-8c6e-9014-df0a8f3b5e25 (at 10.8.6.17@o2ib6) Sep 11 10:28:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7851f52d-5071-abbd-16bd-5eca04911884 (at 10.8.6.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818feafe800, cur 1536686907 expire 1536686757 last 1536686680 Sep 11 10:37:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d5bbb2f3-d359-d4c1-038d-52b7d34607f5 (at 10.8.6.2@o2ib6) Sep 11 10:52:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 98697bfc-793f-4b11-46e7-9a9d15be7a2c (at 10.8.6.19@o2ib6) Sep 11 11:05:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d296a15e-8154-8527-be08-b302d3e42b22 (at 10.8.6.21@o2ib6) Sep 11 11:18:28 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 11:27:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 89ed231a-44d4-d822-879a-f2f1f2f402ed (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa85d400, cur 1536690420 expire 1536690270 last 1536690193 Sep 11 11:35:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5158eccf-14ae-5d94-c749-4edeb98a44a2 (at 10.8.6.8@o2ib6) Sep 11 11:43:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 11 12:01:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4db94d05-0dab-919f-3011-c5b8ea415e50 (at 10.8.6.13@o2ib6) Sep 11 12:02:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6a30dc3-385c-98b8-7125-167ab8dc258a (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880355d02800, cur 1536692560 expire 1536692410 last 1536692333 Sep 11 12:22:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ca4de82a-a28c-8cdd-44d5-f14392933650 (at 10.8.9.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b61431800, cur 1536693721 expire 1536693571 last 1536693494 Sep 11 12:38:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6bfa8ddc-353b-49b6-cd2d-dea5be0ded31 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003c7f5000, cur 1536694697 expire 1536694547 last 1536694470 Sep 11 12:53:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca4de82a-a28c-8cdd-44d5-f14392933650 (at 10.8.9.1@o2ib6) Sep 11 13:21:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 77f4bc87-fc96-8ea2-5a75-a76e0d560061 (at 10.8.9.7@o2ib6) Sep 11 13:27:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 11 13:28:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 26cb7aee-682b-9bfc-1baf-18c40c939d12 (at 10.8.6.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c8a7d1400, cur 1536697703 expire 1536697553 last 1536697476 Sep 11 13:29:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9638f799-7c88-d188-f651-da3d16004a31 (at 10.8.4.34@o2ib6) Sep 11 13:30:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8ee71612-e9fd-8c6e-9014-df0a8f3b5e25 (at 10.8.6.17@o2ib6) Sep 11 13:32:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5158eccf-14ae-5d94-c749-4edeb98a44a2 (at 10.8.6.8@o2ib6) Sep 11 13:35:44 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 13:40:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 11 13:40:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e94c638-464a-d85e-332a-f606c2411391 (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005b4d6000, cur 1536698458 expire 1536698308 last 1536698231 Sep 11 13:40:58 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 11 13:41:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 11 15:29:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fb625a12-5a65-329e-83ae-0031e6af6ee0 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880150eff800, cur 1536704995 expire 1536704845 last 1536704768 Sep 11 15:56:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 11 16:03:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d85dd6b-c92b-660d-fcaa-6da2163a1f77 (at 10.8.1.14@o2ib6) Sep 11 16:10:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c46623c2-a4c4-9fd0-76b1-93132a9d5817 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cb95ebc00, cur 1536707434 expire 1536707284 last 1536707207 Sep 11 16:30:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7684ea1f-c6ca-0559-feee-79839b865673 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8657c000, cur 1536708604 expire 1536708454 last 1536708377 Sep 11 16:31:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 11 16:31:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a229ada3-99e5-45f8-0ba6-7ede88791148 (at 10.8.4.2@o2ib6) Sep 11 16:32:28 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 16:54:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 11 16:55:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f0b87ac0-436f-82e9-0d6d-6e9faee99bf4 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb4a7b000, cur 1536710124 expire 1536709974 last 1536709897 Sep 11 16:59:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Sep 11 17:00:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 000825e5-5a97-ff18-d0fb-3b47c7cf2651 (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88070479f400, cur 1536710402 expire 1536710252 last 1536710175 Sep 11 17:10:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 11 17:17:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9effb4e8-f195-a3eb-e68a-859b1f5dda95 (at 10.8.14.4@o2ib6) Sep 11 17:58:08 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 18:07:17 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 11 18:07:17 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 11 19:41:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d7fd36fb-3e84-f22b-36ad-d6769fa9e6be (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880022481800, cur 1536720074 expire 1536719924 last 1536719847 Sep 11 19:55:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 11 20:11:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 808bbb57-b2f5-2ce1-bbbd-797c6b21fbeb (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800365b8800, cur 1536721882 expire 1536721732 last 1536721655 Sep 11 20:16:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 12 00:09:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0171ffa8-5dfa-870b-bd42-86d181bcf7e4 (at 10.8.4.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005cdcbc00, cur 1536736145 expire 1536735995 last 1536735918 Sep 12 00:09:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 12 04:51:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 67e4f0e5-878d-8504-3926-577e8e817fdc (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880041c1b400, cur 1536753067 expire 1536752917 last 1536752840 Sep 12 04:52:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 12 05:02:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a72417b-7f08-1522-fb54-7be0dbe0a095 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cb95ed400, cur 1536753754 expire 1536753604 last 1536753527 Sep 12 05:03:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 12 05:08:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 12 05:08:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 286c149d-73e8-cff8-5235-7c4e51f42ec8 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210b000, cur 1536754115 expire 1536753965 last 1536753888 Sep 12 06:12:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20ac1145-d922-5fed-fb67-75b204431e5e (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005c333c00, cur 1536757931 expire 1536757781 last 1536757704 Sep 12 06:51:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 723dd7cd-c7d8-fe41-cc61-70d7ebd5b5dc (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f3000, cur 1536760315 expire 1536760165 last 1536760088 Sep 12 09:09:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 046b6f7b-5531-8a41-d540-00485b286155 (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096bf04c00, cur 1536768595 expire 1536768445 last 1536768368 Sep 12 10:01:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e0bb2531-586c-138e-849a-c911f159bd14 (at 10.8.4.5@o2ib6) reconnecting Sep 12 10:01:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Sep 12 10:02:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e0bb2531-586c-138e-849a-c911f159bd14 (at 10.8.4.5@o2ib6) reconnecting Sep 12 10:02:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Sep 12 10:03:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 72e213c3-5090-e3e6-76d5-43e6eaba24ec (at 10.0.2.60@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e75237c00, cur 1536771824 expire 1536771674 last 1536771597 Sep 12 10:03:44 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 12 10:14:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e0bb2531-586c-138e-849a-c911f159bd14 (at 10.8.4.5@o2ib6) reconnecting Sep 12 10:14:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Sep 12 10:25:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7871cc6-9cea-1b22-d22c-c4315ab6141f (at 10.0.2.242@o2ib5) Sep 12 10:26:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7871cc6-9cea-1b22-d22c-c4315ab6141f (at 10.0.2.242@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f63f400, cur 1536773181 expire 1536773031 last 1536772954 Sep 12 10:32:56 oak-md1-s2 kernel: LNet: 464:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.242@o2ib5 version 12/12 incarnation 1536773112714047/1536773549972527 Sep 12 10:32:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7871cc6-9cea-1b22-d22c-c4315ab6141f (at 10.0.2.242@o2ib5) Sep 12 11:31:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 72e213c3-5090-e3e6-76d5-43e6eaba24ec (at 10.0.2.60@o2ib5) Sep 12 11:35:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 72e213c3-5090-e3e6-76d5-43e6eaba24ec (at 10.0.2.60@o2ib5) Sep 12 11:42:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.60@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Sep 12 11:42:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Sep 12 12:03:04 oak-md1-s2 kernel: Lustre: DEBUG MARKER: Wed Sep 12 12:03:04 2018 Sep 12 12:48:44 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 15:01:13 oak-md1-s2 kernel: LNet: 464:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.60@o2ib5 version 12/12 incarnation 1536777283156183/1536789648577976 Sep 12 15:01:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 72e213c3-5090-e3e6-76d5-43e6eaba24ec (at 10.0.2.60@o2ib5) Sep 12 15:03:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eef1eb28-4b06-9d69-1655-76b7ba9e2132 (at 10.0.2.60@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db2f80000, cur 1536789802 expire 1536789652 last 1536789575 Sep 12 15:08:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Sep 12 15:10:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c26648fa-7ebe-2fb6-6200-13f31b24559d (at 10.8.6.30@o2ib6) Sep 12 15:10:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 12 15:11:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 046b6f7b-5531-8a41-d540-00485b286155 (at 10.8.15.3@o2ib6) Sep 12 15:11:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 12 15:12:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d65d6a82-e961-8992-2e5b-f5d98119df47 (at 10.8.14.9@o2ib6) Sep 12 15:12:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 12 15:13:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9effb4e8-f195-a3eb-e68a-859b1f5dda95 (at 10.8.14.4@o2ib6) Sep 12 15:20:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9f35ebc7-dc64-a998-8312-2cf35186f7db (at 10.8.6.18@o2ib6) Sep 12 15:20:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 12 15:39:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a021cb10-bd6a-f341-103d-4d2bd37ef3c4 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800191dbc00, cur 1536791987 expire 1536791837 last 1536791760 Sep 12 15:40:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 12 15:40:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 12 15:42:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 119c4deb-7682-bbf9-8935-043167268535 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880150ef8000, cur 1536792162 expire 1536792012 last 1536791935 Sep 12 15:43:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 496ef220-f8e8-17c7-9b98-62fd400ab473 (at 10.8.4.15@o2ib6) in 168 seconds. I think it's dead, and I am evicting it. exp ffff8800238cb000, cur 1536792238 expire 1536792088 last 1536792070 Sep 12 15:44:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 12 16:28:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7805fa0d-7536-2d07-c3f3-972f8b288d17 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0da8f000, cur 1536794885 expire 1536794735 last 1536794658 Sep 12 16:48:34 oak-md1-s2 kernel: Lustre: 4293:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536796107/real 1536796107] req@ffff881db8844500 x1610627073871040/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536796114 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 12 16:48:34 oak-md1-s2 kernel: Lustre: 4293:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Sep 12 16:48:48 oak-md1-s2 kernel: Lustre: 4293:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536796121/real 1536796121] req@ffff881db8844500 x1610627073871040/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536796128 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 12 16:48:48 oak-md1-s2 kernel: Lustre: 4293:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 12 16:49:09 oak-md1-s2 kernel: Lustre: 4293:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536796142/real 1536796142] req@ffff881db8844500 x1610627073871040/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536796149 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 12 16:49:09 oak-md1-s2 kernel: Lustre: 4293:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 12 16:49:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 16:49:38 oak-md1-s2 kernel: LustreError: 4293:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.18.35@o2ib6) returned error from blocking AST (req@ffff881db8844500 x1610627073871040 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880b4310e400/0x6c00c3e287fb1a1c lrc: 4/0,0 mode: PR/PR res: [0x2000173e0:0x2a50:0x0].0x0 bits 0x13 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.8.18.35@o2ib6 remote: 0xf5c35b83e7e9eb38 expref: 1737 pid: 10581 timeout: 5081270978 lvb_type: 0 Sep 12 16:49:38 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.18.35@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Sep 12 16:51:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9458233-bdae-35ee-792d-b7ccd6133d12 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880359e37400, cur 1536796319 expire 1536796169 last 1536796092 Sep 12 16:52:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 16:53:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3fcd3d46-73d4-48e7-ba80-0e621970fe29 (at 10.8.18.35@o2ib6) in 218 seconds. I think it's dead, and I am evicting it. exp ffff88005b4d7800, cur 1536796395 expire 1536796245 last 1536796177 Sep 12 16:54:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 12 16:55:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 16:56:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f6e4129-7a9d-bc5a-ce6c-427f8002e93f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8d3a000, cur 1536796571 expire 1536796421 last 1536796344 Sep 12 16:57:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 081b1e47-2537-c4b4-7f27-37a90c240691 (at 10.8.15.5@o2ib6) in 190 seconds. I think it's dead, and I am evicting it. exp ffff88070479e800, cur 1536796647 expire 1536796497 last 1536796457 Sep 12 16:58:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 92fb6e94-ee91-72b2-ddb1-d108b4cda850 (at 10.8.18.35@o2ib6) in 158 seconds. I think it's dead, and I am evicting it. exp ffff880046e2ec00, cur 1536796723 expire 1536796573 last 1536796565 Sep 12 16:59:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 17:03:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client abbaea26-3e8b-b3f5-08d6-0764d06c3054 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880359e34400, cur 1536797038 expire 1536796888 last 1536796811 Sep 12 17:24:05 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 17:34:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 17:37:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 17:38:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b4e330e0-c83a-0283-60e1-60f73c232f49 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880052c84c00, cur 1536799097 expire 1536798947 last 1536798870 Sep 12 17:42:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 17:42:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02387da3-7736-6ac2-4969-d5aac4042288 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea84dc400, cur 1536799374 expire 1536799224 last 1536799147 Sep 12 17:55:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 12 17:55:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 12 18:01:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 12 18:01:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 12 18:04:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9638f799-7c88-d188-f651-da3d16004a31 (at 10.8.4.34@o2ib6) Sep 12 18:36:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Sep 12 18:44:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6824c1c6-06e9-621e-44df-29eb02bbe2c6 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c46161000, cur 1536803075 expire 1536802925 last 1536802848 Sep 12 19:00:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0f100744-51a8-97bc-8c69-2d7f3b1b6767 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11861c00, cur 1536804028 expire 1536803878 last 1536803801 Sep 12 19:02:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 12 19:09:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97d95a50-d6b0-3a66-695f-6bb4cac334c6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005c336c00, cur 1536804559 expire 1536804409 last 1536804332 Sep 12 19:09:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 12 19:10:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 19:14:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 19:14:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f6e9340-68c3-07b3-65f4-01c6e600ffc0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880860fa0400, cur 1536804860 expire 1536804710 last 1536804633 Sep 12 19:17:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 57e0f3e4-a9e4-6ce8-00b2-da462ae54769 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806a9e64c00, cur 1536805074 expire 1536804924 last 1536804847 Sep 12 19:28:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 19:31:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d748edf-7de1-b0a6-75c6-664d4e7096a0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005b62c800, cur 1536805914 expire 1536805764 last 1536805687 Sep 12 19:39:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 19:44:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 76fa2cc4-872b-f01c-c8c2-94574cbdeb6b (at 10.0.2.60@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dab515400, cur 1536806679 expire 1536806529 last 1536806452 Sep 12 19:56:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 19:57:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9876462a-a0d8-4fdb-14d8-a180963af60b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f15cb0c00, cur 1536807423 expire 1536807273 last 1536807196 Sep 12 20:02:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 20:03:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a51758c8-3c18-dea4-2364-fa8a8dc617e9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e39800, cur 1536807805 expire 1536807655 last 1536807578 Sep 12 20:07:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0d8ca17-48c8-21a8-656d-b06e1a0c1974 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f03e19800, cur 1536808053 expire 1536807903 last 1536807826 Sep 12 20:09:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 20:15:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c01e1dce-ed4f-08ba-9f97-91fc112254b7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f15cb6400, cur 1536808502 expire 1536808352 last 1536808275 Sep 12 20:17:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 20:21:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 20:22:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fcd6e0c4-b293-38e0-5987-93808ed56058 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c8a7d6800, cur 1536808967 expire 1536808817 last 1536808740 Sep 12 20:26:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3053dd5-0b14-fde8-91f6-4b34b586aba9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea7400, cur 1536809210 expire 1536809060 last 1536808983 Sep 12 20:26:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 20:34:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c83d91ed-94fe-7584-6196-09c89950d632 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880704798400, cur 1536809682 expire 1536809532 last 1536809455 Sep 12 20:56:08 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 21:00:36 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 21:11:02 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 21:11:02 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 12 21:11:45 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 21:11:45 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 12 21:42:02 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:08:54 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:08:58 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:14:58 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:22:22 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:31:43 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:31:43 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 12 22:42:45 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:47:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 825e25d6-46ba-66b5-860a-7df142eb25ba (at 10.8.4.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88067beddc00, cur 1536817655 expire 1536817505 last 1536817428 Sep 12 22:50:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 22:50:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d56ba079-10cb-d3a4-5f75-9fa6ae69067c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ffaffb800, cur 1536817833 expire 1536817683 last 1536817606 Sep 12 22:51:04 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:51:04 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Sep 12 22:53:27 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:53:27 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Sep 12 22:54:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 22:55:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 483e310d-47e6-43a9-9762-608c836a53e1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f2ae800, cur 1536818130 expire 1536817980 last 1536817903 Sep 12 22:55:57 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:56:28 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:58:04 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 22:58:04 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 12 22:59:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client daf6af47-9ed3-b9cc-ff24-989e699c1c24 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c73eaa400, cur 1536818399 expire 1536818249 last 1536818172 Sep 12 23:01:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 23:05:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e8e2308-0ebf-d302-ca08-9360c072bd1b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88060de92400, cur 1536818731 expire 1536818581 last 1536818504 Sep 12 23:08:21 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:18:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 23:21:45 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:21:45 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 12 23:23:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05662087-8c15-39d0-c139-28d56e0e7ba1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880373438400, cur 1536819790 expire 1536819640 last 1536819563 Sep 12 23:23:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 23:27:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb8cb981-66f5-c09b-ab83-585414a0ed5c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd633d800, cur 1536820040 expire 1536819890 last 1536819813 Sep 12 23:29:25 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:29:25 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 12 23:33:10 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:33:10 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Sep 12 23:36:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 12 23:37:39 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:37:39 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 12 23:38:50 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:38:50 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 12 23:43:02 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:43:02 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 24 previous similar messages Sep 12 23:56:06 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 12 23:56:06 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 13 00:01:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 000665e4-6d4c-b134-29b1-0da345d73186 (at 10.8.6.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092ea03400, cur 1536822098 expire 1536821948 last 1536821871 Sep 13 00:06:17 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 00:06:17 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 13 00:12:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e62e7533-1106-7bae-e0ae-cddffbed489e (at 10.8.23.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b63e32400, cur 1536822778 expire 1536822628 last 1536822551 Sep 13 00:49:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71ab68e8-cec2-c683-ada2-aa65d070e01f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e80ef7400, cur 1536824990 expire 1536824840 last 1536824763 Sep 13 00:58:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 01:00:49 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 01:12:47 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 01:12:47 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 13 01:14:08 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 01:14:08 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 13 01:15:27 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 01:25:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d70a86d5-376d-2071-3d95-c94b11eea046 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11860400, cur 1536827103 expire 1536826953 last 1536826876 Sep 13 01:25:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 13 01:27:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 01:35:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 01:35:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f1e6aa49-489e-d836-1731-1f8000b35a7b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb84800, cur 1536827737 expire 1536827587 last 1536827510 Sep 13 01:45:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 54616537-9b8a-f9f3-ca99-661b4037a9fc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8c9400, cur 1536828314 expire 1536828164 last 1536828087 Sep 13 01:46:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 02:12:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52bc0b8a-cc3e-e060-18f0-5f2105ec64ef (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0da8d400, cur 1536829928 expire 1536829778 last 1536829701 Sep 13 02:14:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 02:15:28 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 02:15:28 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 13 02:31:27 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 02:35:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f98b382-b7a5-2779-0367-f570b221345d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582305c00, cur 1536831331 expire 1536831181 last 1536831104 Sep 13 02:46:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 02:59:00 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 02:59:00 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 13 03:02:52 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 03:50:53 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 03:50:53 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 13 04:02:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f63cd7db-8a89-426e-53aa-3a9a5a421edc (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805adfae800, cur 1536836527 expire 1536836377 last 1536836300 Sep 13 04:11:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 04:14:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 04:15:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bbc0a74f-53ce-0d84-58e9-0f407c5e2278 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001332a000, cur 1536837326 expire 1536837176 last 1536837099 Sep 13 04:42:21 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 04:48:57 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 05:44:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f8b83694-92c5-06e7-91aa-bf9c5a792184 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099ad21800, cur 1536842692 expire 1536842542 last 1536842465 Sep 13 05:53:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 06:01:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4858c3c8-8690-72c0-88da-e9f3de7454b3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803b5648800, cur 1536843713 expire 1536843563 last 1536843486 Sep 13 06:01:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 06:19:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ecf8a97a-181a-fb13-e0b2-a08f9b1ab09a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803b564dc00, cur 1536844744 expire 1536844594 last 1536844517 Sep 13 06:19:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 06:25:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cacbc2f4-f0ae-bc4b-cded-b83d9ca4c5b2 (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c8a7d6400, cur 1536845148 expire 1536844998 last 1536844921 Sep 13 06:27:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1938bfdb-47db-0d0f-5160-3b9efcd8d02d (at 10.8.18.35@o2ib6) in 161 seconds. I think it's dead, and I am evicting it. exp ffff880e19c47800, cur 1536845224 expire 1536845074 last 1536845063 Sep 13 06:27:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 06:50:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 06:50:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4c47a0da-6c9a-9281-8282-300d507f9cc2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bebfa8c00, cur 1536846644 expire 1536846494 last 1536846417 Sep 13 08:12:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 13 08:14:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 13 08:14:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 13 08:15:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a2e6848-dfdf-71df-199c-a16000147612 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803cb636800, cur 1536851747 expire 1536851597 last 1536851520 Sep 13 08:25:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 09a9e5e5-dc43-8a42-ced0-ab71a4cb5e18 (at 10.12.4.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200bed4c00, cur 1536852324 expire 1536852174 last 1536852097 Sep 13 08:27:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c26648fa-7ebe-2fb6-6200-13f31b24559d (at 10.8.6.30@o2ib6) Sep 13 08:30:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cdcd05c3-54f5-8748-3ca6-a470bb0610fb (at 10.8.4.9@o2ib6) Sep 13 08:32:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4883f597-a046-b191-25f2-506981351c90 (at 10.8.6.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc3efc000, cur 1536852773 expire 1536852623 last 1536852546 Sep 13 08:39:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e6af7493-5c77-067f-6cee-37f589a3ba62 (at 10.8.2.19@o2ib6) Sep 13 08:49:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 13 09:10:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49787486-f557-40bc-a774-469550a868d1 (at 10.12.4.48@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039012000, cur 1536855020 expire 1536854870 last 1536854793 Sep 13 09:10:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42ddaac4-4249-5052-00c0-3d0a27940cdb (at 10.0.2.224@o2ib5) Sep 13 09:11:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5becf9f-11f3-8959-6470-225ad411ffff (at 10.8.11.22@o2ib6) in 154 seconds. I think it's dead, and I am evicting it. exp ffff88001494f400, cur 1536855096 expire 1536854946 last 1536854942 Sep 13 09:14:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 13 09:15:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42ddaac4-4249-5052-00c0-3d0a27940cdb (at 10.0.2.224@o2ib5) Sep 13 09:41:23 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.224@o2ib5 version 12/12 incarnation 1536855319188021/1536856857634552 Sep 13 09:41:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42ddaac4-4249-5052-00c0-3d0a27940cdb (at 10.0.2.224@o2ib5) Sep 13 10:05:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 13 10:06:28 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 10:07:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 44f8a0b6-4c79-8b83-7ea9-94110b49162b (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88006735c000, cur 1536858428 expire 1536858278 last 1536858201 Sep 13 10:24:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c26648fa-7ebe-2fb6-6200-13f31b24559d (at 10.8.6.30@o2ib6) Sep 13 10:28:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8b028d57-c045-6786-fcf0-f9265eff00d3 (at 10.8.6.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a86578c00, cur 1536859717 expire 1536859567 last 1536859490 Sep 13 10:31:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ef6e25c-72f9-c5f0-e3de-bb0905350ad8 (at 10.12.4.65@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fe26400, cur 1536859906 expire 1536859756 last 1536859679 Sep 13 10:38:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 10:38:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 546b9887-bbec-a629-d715-74eccfab575d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88145afee400, cur 1536860283 expire 1536860133 last 1536860056 Sep 13 10:42:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5475013d-21b0-eafe-66db-2bdce134a3c2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c46164400, cur 1536860541 expire 1536860391 last 1536860314 Sep 13 10:51:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 10:58:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 10:58:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99f1bf74-b4bd-0384-cff1-3ed136d4eba0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bafc0000, cur 1536861519 expire 1536861369 last 1536861292 Sep 13 11:03:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 11:03:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74feb181-a515-fa6c-8222-32cd5b15cd97 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e150cbc00, cur 1536861814 expire 1536861664 last 1536861587 Sep 13 11:05:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4340ff4d-7982-f3ce-a4c5-83270076899c (at 10.9.101.39@o2ib4) Sep 13 11:06:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4340ff4d-7982-f3ce-a4c5-83270076899c (at 10.9.101.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a37ac00, cur 1536862000 expire 1536861850 last 1536861773 Sep 13 11:19:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ae7961f1-c27d-d378-44be-730fb7c7cbd3 (at 10.12.4.65@o2ib) Sep 13 11:20:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f32e1a5-eb31-6a20-2273-e3261970c3b9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bafc0400, cur 1536862808 expire 1536862658 last 1536862581 Sep 13 11:31:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 11:32:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4340ff4d-7982-f3ce-a4c5-83270076899c (at 10.9.101.39@o2ib4) Sep 13 11:33:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 399a1000-92f6-6e70-a7bc-25491dc4ccf1 (at 10.9.101.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e38c00, cur 1536863626 expire 1536863476 last 1536863399 Sep 13 11:35:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3eebc8c8-119b-850b-9382-66123d9d7e5f (at 10.8.18.35@o2ib6) in 209 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a7c00, cur 1536863702 expire 1536863552 last 1536863493 Sep 13 11:43:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09a9e5e5-dc43-8a42-ced0-ab71a4cb5e18 (at 10.12.4.39@o2ib) Sep 13 11:43:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49787486-f557-40bc-a774-469550a868d1 (at 10.12.4.48@o2ib) Sep 13 11:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 11:49:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 017a2337-0f00-3490-56f7-fb1941748f8b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000283fc00, cur 1536864548 expire 1536864398 last 1536864321 Sep 13 11:49:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 11:52:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3aab68ba-2531-c11a-26c4-0152ae49af03 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d4400, cur 1536864724 expire 1536864574 last 1536864497 Sep 13 11:53:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5185e8a7-12da-ef5c-6ff9-71bc9c430db2 (at 10.8.18.35@o2ib6) in 213 seconds. I think it's dead, and I am evicting it. exp ffff880d773e2c00, cur 1536864800 expire 1536864650 last 1536864587 Sep 13 11:56:45 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 12:02:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 12:05:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e636a277-a1b2-6eca-2514-5a232c5110c5 (at 10.8.6.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7bed000, cur 1536865556 expire 1536865406 last 1536865329 Sep 13 12:07:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 12:12:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 12:12:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f5cebaf-0838-32f0-5429-279cb991fa54 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc48ce800, cur 1536865949 expire 1536865799 last 1536865722 Sep 13 12:12:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 13 12:16:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2fd8925d-a49a-6ee7-8990-3ca3fc211609 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d5800, cur 1536866196 expire 1536866046 last 1536865969 Sep 13 12:16:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 12:19:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ddb55fc-2f02-ff97-c3da-9b85ffb722f7 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aca2c7400, cur 1536866383 expire 1536866233 last 1536866156 Sep 13 12:34:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79fe6e98-e151-b663-595c-d3465ca6f691 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0e1e4400, cur 1536867259 expire 1536867109 last 1536867032 Sep 13 12:42:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb668524-cacf-35e2-d64d-5376b61b0f43 (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807acece000, cur 1536867735 expire 1536867585 last 1536867508 Sep 13 12:43:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 12:47:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 12:48:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a083e101-9a77-90ae-1d9d-3f69fed17770 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099ad23800, cur 1536868112 expire 1536867962 last 1536867885 Sep 13 13:46:57 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 13:54:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Sep 13 13:54:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.60@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Sep 13 13:54:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6acc0c1d-c52f-aaec-09e8-493de0459c0f (at 10.0.2.61@o2ib5) Sep 13 13:55:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.60@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Sep 13 13:55:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Sep 13 13:56:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[60-61]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Sep 13 13:56:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Sep 13 14:06:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6acc0c1d-c52f-aaec-09e8-493de0459c0f (at 10.0.2.61@o2ib5) Sep 13 14:12:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3189a414-d873-809f-2811-3f7082533ab4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8cb000, cur 1536873175 expire 1536873025 last 1536872948 Sep 13 14:13:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 14:25:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f8655401-4ffe-ba2d-a729-d1241d632686 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800280d5000, cur 1536873947 expire 1536873797 last 1536873720 Sep 13 14:27:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 13 14:33:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38e554b6-052f-3d5f-c05b-b7c33c8b57d9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c8a7d6c00, cur 1536874426 expire 1536874276 last 1536874199 Sep 13 14:35:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 14:53:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cead9ba3-efcd-da77-45d2-0f853f20869f (at 10.8.4.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc3efd400, cur 1536875593 expire 1536875443 last 1536875366 Sep 13 15:04:23 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 15:32:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 64887e3b-9e10-8f96-8878-f8ebcbf3501f (at 10.0.2.61@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd65fc00, cur 1536877965 expire 1536877815 last 1536877738 Sep 13 15:40:50 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 15:55:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 424d9e99-ff53-1c17-9f63-ab6dcd55003a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e4400, cur 1536879336 expire 1536879186 last 1536879109 Sep 13 16:05:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 16:07:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b531d800-1488-76a4-03aa-5c1f4cd9c33b (at 10.0.2.67@o2ib5) Sep 13 16:09:28 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.67@o2ib5 version 12/12 incarnation 1536880011421155/1536880142089691 Sep 13 16:09:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b531d800-1488-76a4-03aa-5c1f4cd9c33b (at 10.0.2.67@o2ib5) Sep 13 16:15:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[60-61]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Sep 13 16:15:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Sep 13 16:17:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[66-67]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Sep 13 16:17:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Sep 13 16:17:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5b16b733-c469-1f53-4966-baba53831751 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b5f378400, cur 1536880633 expire 1536880483 last 1536880406 Sep 13 16:18:31 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.67@o2ib5 version 12/12 incarnation 1536880142089691/1536880685230091 Sep 13 16:18:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b531d800-1488-76a4-03aa-5c1f4cd9c33b (at 10.0.2.67@o2ib5) Sep 13 16:19:08 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 16:24:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 16:25:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 269e0e14-def8-4795-b62a-7c794814a96a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a596b5400, cur 1536881111 expire 1536880961 last 1536880884 Sep 13 16:29:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 16:30:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3365db68-bc28-af26-c038-8c6a85aafdde (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e902ba000, cur 1536881425 expire 1536881275 last 1536881198 Sep 13 16:31:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 13 17:12:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 13 17:14:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5467f5bc-222e-a4de-33d2-11a9032c85f0 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88057a351c00, cur 1536884059 expire 1536883909 last 1536883832 Sep 13 17:32:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 13 17:36:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 95d3d72c-8035-09f5-a5ad-3dea6e11befd (at 10.8.26.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802c8b63000, cur 1536885361 expire 1536885211 last 1536885134 Sep 13 17:39:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7737b6c5-b9fd-ad2f-3fc6-e52ae308bc51 (at 10.8.14.5@o2ib6) Sep 13 17:39:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 13 17:41:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cdcd05c3-54f5-8748-3ca6-a470bb0610fb (at 10.8.4.9@o2ib6) Sep 13 17:42:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 61e1628c-d4fe-66fb-68e7-6900ad1d754c (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfefc4800, cur 1536885748 expire 1536885598 last 1536885521 Sep 13 17:43:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 13 17:54:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 95d3d72c-8035-09f5-a5ad-3dea6e11befd (at 10.8.26.29@o2ib6) Sep 13 17:58:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 13 17:58:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7108ca3f-9a16-2c82-6991-8e24ae8ad4f7 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f03e1bc00, cur 1536886734 expire 1536886584 last 1536886507 Sep 13 18:15:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1360e146-43e4-9f4a-0e06-3594a04e9480 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88020cea0800, cur 1536887748 expire 1536887598 last 1536887521 Sep 13 18:16:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 13 18:30:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d602bfbd-5ebb-ccb8-a4fe-c4cdcb6bfc8d (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aedf7e000, cur 1536888638 expire 1536888488 last 1536888411 Sep 13 18:30:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 13 18:47:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bd9f2af-18d1-8666-8260-3bad3ba78584 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea2000, cur 1536889667 expire 1536889517 last 1536889440 Sep 13 18:48:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 13 18:54:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 13 19:07:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e41335ba-9eb3-42ed-c5c9-0dba1566a72f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e3c800, cur 1536890849 expire 1536890699 last 1536890622 Sep 13 19:08:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 19:11:53 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 19:12:11 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 19:19:28 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 13 19:45:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4369c3f-1ca5-b9c7-e92e-2b81a126c246 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b720400, cur 1536893154 expire 1536893004 last 1536892927 Sep 13 19:46:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f4369c3f-1ca5-b9c7-e92e-2b81a126c246 (at 10.8.11.20@o2ib6) Sep 13 20:18:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 13 20:55:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 20:56:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e2cb190-eb54-d84f-6fc2-49127c2c9d35 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4a35dc00, cur 1536897369 expire 1536897219 last 1536897142 Sep 13 21:12:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 13 21:12:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6980fb20-b41f-2c74-1864-0e92b898f5e0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3aff800, cur 1536898354 expire 1536898204 last 1536898127 Sep 13 21:28:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0a68485a-b7c1-b6b6-93e7-d0dcac3ee93f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a5f77b400, cur 1536899288 expire 1536899138 last 1536899061 Sep 13 21:31:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 21:37:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4c681e72-3b5f-e2fa-bdb4-f17f1b089c36 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039237c00, cur 1536899849 expire 1536899699 last 1536899622 Sep 13 22:01:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 22:04:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 22:05:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1be47ba2-12d9-b8ab-ac66-3da13deb7292 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c7fa21000, cur 1536901537 expire 1536901387 last 1536901310 Sep 13 22:20:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b2a5fa66-7e85-907c-146d-3e080d3db188 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039f59f000, cur 1536902446 expire 1536902296 last 1536902219 Sep 13 22:32:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 22:50:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 13 22:52:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9e6cc65-272a-188f-1fec-ec65ab2bd7d9 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c93ff3400, cur 1536904359 expire 1536904209 last 1536904132 Sep 13 23:09:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 13 23:10:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client feeb9092-ce09-4dbb-ed30-5c7f36411ed6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e20d7e400, cur 1536905400 expire 1536905250 last 1536905173 Sep 13 23:13:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 13 23:25:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a798ee7-b6a7-09f5-354f-8dd171dcaa4b (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb87400, cur 1536906322 expire 1536906172 last 1536906095 Sep 13 23:48:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 13 23:56:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02adaed3-d69e-e948-e22d-47b86e08011d (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880125571400, cur 1536908169 expire 1536908019 last 1536907942 Sep 14 00:23:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8b1bcfbb-e727-8508-1299-4060378ceb09 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eaa67ac00, cur 1536909804 expire 1536909654 last 1536909577 Sep 14 00:26:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 14 00:34:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90949fa5-75b1-e8f0-f24a-3a4c3a0fb03e (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0da8a800, cur 1536910490 expire 1536910340 last 1536910263 Sep 14 00:57:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 14 01:05:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f97ad16-c3a8-d2da-1039-dcb03cadc91f (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f3d886400, cur 1536912333 expire 1536912183 last 1536912106 Sep 14 01:30:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 14 01:34:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 14 01:35:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d67a7c4e-c5a5-2b24-bf3e-2674df0be24c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880420555c00, cur 1536914122 expire 1536913972 last 1536913895 Sep 14 01:36:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0b04376e-e2e2-059f-f15e-cad74b640f44 (at 10.8.15.5@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff8802fac1bc00, cur 1536914198 expire 1536914048 last 1536914027 Sep 14 01:49:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 14 01:56:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 14 02:04:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51b3d156-7b03-c8f1-c75a-d2e35622a840 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801919d8c00, cur 1536915846 expire 1536915696 last 1536915619 Sep 14 02:21:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 14 02:24:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 14 02:32:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 871d57a1-8c2e-02dd-57a0-442ffd34dc77 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fbc5be800, cur 1536917524 expire 1536917374 last 1536917297 Sep 14 02:53:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 14 02:55:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 14 03:16:39 oak-md1-s2 kernel: Lustre: 10530:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 03:25:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 14 03:33:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 513be8e1-5da9-0a55-2993-67e6a721a58a (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cba381400, cur 1536921194 expire 1536921044 last 1536920967 Sep 14 03:56:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 14 03:59:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 610af179-123f-79eb-d5e1-07cc66387305 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c8a7d6000, cur 1536922797 expire 1536922647 last 1536922570 Sep 14 04:04:59 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 04:05:00 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 04:05:01 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 04:05:01 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 14 04:10:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 14 04:20:02 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 04:20:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65a4ab09-6afc-bf04-a950-1755cbc53f39 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f59d86c00, cur 1536924025 expire 1536923875 last 1536923798 Sep 14 04:21:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 14 04:30:18 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 05:02:02 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 05:02:03 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 06:08:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e5dc00, cur 1536930501 expire 1536930351 last 1536930274 Sep 14 06:21:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0cb190f-3650-3686-e7aa-45c94cec4a32 (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb86800, cur 1536931303 expire 1536931153 last 1536931076 Sep 14 06:24:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 06:48:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dddc5f72-d796-4dda-b993-887a2930284c (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880effd15000, cur 1536932921 expire 1536932771 last 1536932694 Sep 14 06:48:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 07:05:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 07:14:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 87a3e9b6-1ef0-a5ae-9b84-cbe841dc71ca (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880806057800, cur 1536934476 expire 1536934326 last 1536934249 Sep 14 07:25:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 07:35:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 07:35:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5d600d96-e8ca-e3ad-6525-ac0ac5cddb4f (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809eb420c00, cur 1536935730 expire 1536935580 last 1536935503 Sep 14 07:45:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 07:45:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8cdf3d7e-e51a-4eb9-a53b-dd4128186f12 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003f6c8c00, cur 1536936356 expire 1536936206 last 1536936129 Sep 14 07:56:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 07:56:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cc2477e9-4514-06b3-f71c-c963f0f3fd09 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880941cd3c00, cur 1536937014 expire 1536936864 last 1536936787 Sep 14 08:05:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4966203-761f-6dc5-9911-bf9e91bbb908 (at 10.8.13.24@o2ib6) Sep 14 08:05:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Sep 14 08:05:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 08:11:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e5505c4f-9546-a0c4-881b-90da24875770 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee0569c00, cur 1536937886 expire 1536937736 last 1536937659 Sep 14 08:11:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bccf629c-900e-c4a9-171a-d70129138265 (at 10.8.14.1@o2ib6) Sep 14 08:11:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 08:30:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 08:38:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 08:39:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e50a2e38-a835-9b8a-b7fc-b13bda896a0a (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881843ed5000, cur 1536939573 expire 1536939423 last 1536939346 Sep 14 09:42:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 003bfd88-919c-3745-858b-eadb4f8ad9d1 (at 10.12.4.55@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be00800, cur 1536943355 expire 1536943205 last 1536943128 Sep 14 10:09:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ecd5a14-5867-79c1-8dc0-a713c6faf6dd (at 10.9.102.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565c8800, cur 1536944977 expire 1536944827 last 1536944750 Sep 14 10:30:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ecd5a14-5867-79c1-8dc0-a713c6faf6dd (at 10.9.102.33@o2ib4) Sep 14 10:34:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ef2087a-6b13-bd77-8cb8-2223fda4f9ba (at 10.8.8.27@o2ib6) Sep 14 11:23:26 oak-md1-s2 kernel: Lustre: 10555:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 11:31:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7fd0b240-6bd2-b1b0-de0c-d203b8d0be0f (at 10.8.8.37@o2ib6) Sep 14 11:31:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b40de99f-07a2-e9cc-5fe3-9bcc5ca1cb65 (at 10.8.1.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810392fac00, cur 1536949919 expire 1536949769 last 1536949692 Sep 14 11:31:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 11:32:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 84b30ff3-488e-6243-89b7-d40f1aed2c6e (at 10.8.18.32@o2ib6) Sep 14 11:32:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 951eac6e-f36a-84ad-8654-d42ea1256df5 (at 10.8.18.31@o2ib6) Sep 14 11:32:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4cd85656-cf72-67a5-e479-6a1921e887c6 (at 10.8.1.1@o2ib6) Sep 14 11:33:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 951eac6e-f36a-84ad-8654-d42ea1256df5 (at 10.8.18.31@o2ib6) in 208 seconds. I think it's dead, and I am evicting it. exp ffff8802c8b65c00, cur 1536949995 expire 1536949845 last 1536949787 Sep 14 11:33:15 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 11:33:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b60f6c6-8090-7a5d-9f8c-5e8264218b67 (at 10.8.26.36@o2ib6) Sep 14 11:33:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 11:33:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 14 11:33:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 11:34:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4730725b-dee7-2762-9268-4ecf9882c03e (at 10.9.106.50@o2ib4) Sep 14 11:34:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 11:34:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc4545df-0f85-4787-eae8-4f37b6959728 (at 10.8.18.15@o2ib6) in 211 seconds. I think it's dead, and I am evicting it. exp ffff88019e613c00, cur 1536950071 expire 1536949921 last 1536949860 Sep 14 11:34:31 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 11:35:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f34d0df3-fada-277f-a620-de8260bc2a3c (at 10.9.105.21@o2ib4) in 221 seconds. I think it's dead, and I am evicting it. exp ffff88101b68b800, cur 1536950147 expire 1536949997 last 1536949926 Sep 14 11:35:47 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 11:35:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca2911d7-e15b-8887-321a-5c3bb16e7abb (at 10.8.10.22@o2ib6) Sep 14 11:35:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 11:37:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client caf800bb-b552-335d-d907-fb72a3dffd79 (at 10.9.105.23@o2ib4) in 210 seconds. I think it's dead, and I am evicting it. exp ffff8808cf36a000, cur 1536950223 expire 1536950073 last 1536950013 Sep 14 11:37:03 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 11:37:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a02d950-123b-9a4e-034a-363f1f622796 (at 10.9.106.54@o2ib4) Sep 14 11:37:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 11:38:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5a02d950-123b-9a4e-034a-363f1f622796 (at 10.9.106.54@o2ib4) in 187 seconds. I think it's dead, and I am evicting it. exp ffff88030ca4a800, cur 1536950299 expire 1536950149 last 1536950112 Sep 14 11:38:19 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 11:39:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ecba07e1-c81f-7943-7f37-98c6a4b1cbcf (at 10.9.107.9@o2ib4) in 211 seconds. I think it's dead, and I am evicting it. exp ffff88203a3e2400, cur 1536950375 expire 1536950225 last 1536950164 Sep 14 11:40:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d342f90e-597c-3ec6-8e55-42583e98e20d (at 10.9.107.31@o2ib4) Sep 14 11:40:14 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 14 11:40:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7fec498-8db3-3e88-abdc-898065f3defa (at 10.9.106.55@o2ib4) in 162 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e5a000, cur 1536950451 expire 1536950301 last 1536950289 Sep 14 11:40:51 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 11:42:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa0a1440-c34e-7f04-afaf-afb7e9606852 (at 10.9.107.36@o2ib4) in 222 seconds. I think it's dead, and I am evicting it. exp ffff88203a2f6400, cur 1536950527 expire 1536950377 last 1536950305 Sep 14 11:42:07 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 11:43:38 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536950611/real 1536950611] req@ffff880039971200 x1610627173148928/t0(0) o104->oak-MDT0000@10.8.18.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536950618 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 14 11:43:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 09db39ab-ab74-c18d-09a2-ed0d89f6642c (at 10.8.1.14@o2ib6) reconnecting Sep 14 11:43:38 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 14 11:43:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c93524c0-8a4e-0ae0-d3a7-62de0b9dd711 (at 10.8.27.31@o2ib6) reconnecting Sep 14 11:43:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client b20cddaa-5c85-20ea-9d99-1fe05c59d45f (at 10.8.27.27@o2ib6) reconnecting Sep 14 11:43:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client abfe1850-8d93-1bc2-454d-b1d26ce9a537 (at 10.8.17.15@o2ib6) reconnecting Sep 14 11:43:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 11:43:46 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1536950618/real 1536950611] req@ffff880039971200 x1610627173148928/t0(0) o104->oak-MDT0000@10.8.18.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536950625 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 11:43:46 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Sep 14 11:43:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client b3012777-8296-9224-a263-753cbbd44147 (at 10.8.27.32@o2ib6) reconnecting Sep 14 11:43:51 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 11:43:54 oak-md1-s2 kernel: Lustre: 10665:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536950627/real 1536950627] req@ffff881863e11e00 x1610627173150160/t0(0) o104->oak-MDT0000@10.9.102.7@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536950634 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 11:43:54 oak-md1-s2 kernel: Lustre: 10665:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Sep 14 11:44:16 oak-md1-s2 kernel: Lustre: 10665:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536950648/real 1536950648] req@ffff881863e11e00 x1610627173150160/t0(0) o104->oak-MDT0000@10.9.102.7@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536950655 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 11:44:16 oak-md1-s2 kernel: Lustre: 10665:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Sep 14 11:44:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 09db39ab-ab74-c18d-09a2-ed0d89f6642c (at 10.8.1.14@o2ib6) reconnecting Sep 14 11:44:28 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 11:44:30 oak-md1-s2 kernel: LustreError: 10665:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.102.7@o2ib4) returned error from blocking AST (req@ffff881863e11e00 x1610627173150160 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880024679600/0x6c00c3e689e1abc4 lrc: 4/0,0 mode: PR/PR res: [0x200016f82:0x10cb:0x0].0x0 bits 0x1b rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.9.102.7@o2ib4 remote: 0x4c1c38ca5587cb3c expref: 10 pid: 10572 timeout: 5235763095 lvb_type: 0 Sep 14 11:44:30 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.102.7@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Sep 14 11:44:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 498189b8-2b20-0c96-4790-65b2afa14f59 (at 10.9.106.48@o2ib4) Sep 14 11:44:31 oak-md1-s2 kernel: Lustre: Skipped 56 previous similar messages Sep 14 11:44:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3e98c37-6c62-f76f-b4c5-de4c69d99884 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a978ff400, cur 1536950684 expire 1536950534 last 1536950457 Sep 14 11:44:44 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 11:44:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c93524c0-8a4e-0ae0-d3a7-62de0b9dd711 (at 10.8.27.31@o2ib6) reconnecting Sep 14 11:44:55 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 11:45:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c74f4773-a574-536e-a978-ad56925c5fba (at 10.8.27.25@o2ib6) reconnecting Sep 14 11:45:27 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 14 11:49:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ff4da015-a609-2038-fc6e-6ccc7cf5c6b6 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bef732c00, cur 1536950975 expire 1536950825 last 1536950748 Sep 14 11:49:35 oak-md1-s2 kernel: Lustre: Skipped 165 previous similar messages Sep 14 11:53:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6c9c599f-7ae2-af46-7c69-06f747612dac (at 10.8.18.14@o2ib6) Sep 14 11:53:44 oak-md1-s2 kernel: Lustre: Skipped 51 previous similar messages Sep 14 12:06:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0f6a31c-cba2-90d5-e0fb-013d0247ba26 (at 10.9.107.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880180b5a000, cur 1536951968 expire 1536951818 last 1536951741 Sep 14 12:06:08 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 12:09:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 08516d89-7dca-2e5a-8b13-c3d41b2b439d (at 10.9.105.45@o2ib4) Sep 14 12:09:16 oak-md1-s2 kernel: Lustre: Skipped 47 previous similar messages Sep 14 12:14:04 oak-md1-s2 kernel: LustreError: 10556:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.102.19@o2ib4) returned error from blocking AST (req@ffff88072c507b00 x1610627174573504 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880ef80e6600/0x6c00c3e68a582fc3 lrc: 4/0,0 mode: PR/PR res: [0x20001626e:0x163d1:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.9.102.19@o2ib4 remote: 0x276a21d4ded32312 expref: 22268 pid: 4546 timeout: 5237537171 lvb_type: 0 Sep 14 12:14:04 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.102.19@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Sep 14 12:14:04 oak-md1-s2 kernel: LustreError: 10536:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881b5627f500 x1610627174573904/t0(0) o104->oak-MDT0000@10.9.102.19@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 14 12:14:04 oak-md1-s2 kernel: LustreError: 10536:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Sep 14 12:14:11 oak-md1-s2 kernel: Lustre: 10579:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536952444/real 1536952444] req@ffff880becbb3c00 x1610627174573472/t0(0) o104->oak-MDT0000@10.8.2.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536952451 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 14 12:14:11 oak-md1-s2 kernel: Lustre: 10579:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Sep 14 12:14:18 oak-md1-s2 kernel: Lustre: 10579:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536952451/real 1536952451] req@ffff880becbb3c00 x1610627174573472/t0(0) o104->oak-MDT0000@10.8.2.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536952458 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 12:14:32 oak-md1-s2 kernel: Lustre: 10579:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536952465/real 1536952465] req@ffff880becbb3c00 x1610627174573472/t0(0) o104->oak-MDT0000@10.8.2.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536952472 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 12:14:32 oak-md1-s2 kernel: Lustre: 10579:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 14 12:14:53 oak-md1-s2 kernel: Lustre: 10579:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536952486/real 1536952486] req@ffff880becbb3c00 x1610627174573472/t0(0) o104->oak-MDT0000@10.8.2.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1536952493 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 12:14:53 oak-md1-s2 kernel: Lustre: 10579:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 12:18:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93449d4b-2a14-baac-8aeb-2e675cde5854 (at 10.8.13.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85a8a000, cur 1536952730 expire 1536952580 last 1536952503 Sep 14 12:18:50 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Sep 14 12:19:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 93449d4b-2a14-baac-8aeb-2e675cde5854 (at 10.8.13.3@o2ib6) Sep 14 12:19:37 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Sep 14 12:31:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4171fa08-f56b-664b-748d-3878cc20a1f3 (at 10.8.10.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103a058800, cur 1536953478 expire 1536953328 last 1536953251 Sep 14 12:31:18 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 14 12:31:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 33ec7d14-30b3-1a37-08d4-0b6b1c687758 (at 10.8.25.25@o2ib6) Sep 14 12:31:47 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 14 12:44:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea5140d5-c575-b971-348f-082c09ee870c (at 10.8.23.22@o2ib6) Sep 14 12:44:55 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 12:45:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea5140d5-c575-b971-348f-082c09ee870c (at 10.8.23.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd7a7d000, cur 1536954353 expire 1536954203 last 1536954126 Sep 14 12:45:53 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 14 12:56:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9425f16-111c-0989-b4a7-95612c24ae12 (at 10.8.29.1@o2ib6) Sep 14 12:56:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 13:05:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cac9a134-7469-7012-937a-4cda121773f0 (at 10.8.22.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103a9a4c00, cur 1536955536 expire 1536955386 last 1536955309 Sep 14 13:05:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 13:08:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b15183a6-3a40-f2ab-391b-493af7b7ad4f (at 10.8.6.36@o2ib6) Sep 14 13:08:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 13:20:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a5dbd5f-6274-7c5c-33aa-0775c83c75f9 (at 10.9.101.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e58400, cur 1536956430 expire 1536956280 last 1536956203 Sep 14 13:20:30 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 13:25:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6d4fae06-dc91-78cc-ec54-728aff26fa3d (at 10.9.107.26@o2ib4) Sep 14 13:25:18 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 13:32:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bee5548d-becb-3e18-2501-11666f7a6bf6 (at 10.8.17.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a76ab9400, cur 1536957144 expire 1536956994 last 1536956917 Sep 14 13:32:24 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 13:51:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b8b40ce8-0e35-0947-d1cb-c4e2b1387484 (at 10.9.115.4@o2ib4) Sep 14 13:51:50 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 13:53:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e300380-9ce4-1419-c6a9-495ffc2bcaf9 (at 10.8.26.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fe27400, cur 1536958387 expire 1536958237 last 1536958160 Sep 14 13:53:07 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 13:57:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30eb29f6-d240-7fa7-3f33-284e9d861055 (at 10.8.30.24@o2ib6) Sep 14 13:57:09 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 13:58:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30eb29f6-d240-7fa7-3f33-284e9d861055 (at 10.8.30.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e9a7ba000, cur 1536958681 expire 1536958531 last 1536958454 Sep 14 13:58:01 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 13:59:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0171ba34-2319-29c4-ac03-921fdd8d3203 (at 10.9.105.41@o2ib4) Sep 14 14:00:41 oak-md1-s2 kernel: LustreError: 10649:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.41@o2ib4) returned error from blocking AST (req@ffff881ec1921b00 x1610627180061152 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881452b74e00/0x6c00c3e6a2eb5521 lrc: 4/0,0 mode: PR/PR res: [0x20001621e:0x1832a:0x0].0x0 bits 0x13 rrc: 38 type: IBT flags: 0x60200400000020 nid: 10.9.105.41@o2ib4 remote: 0xe1d2dac36dfff790 expref: 137 pid: 10568 timeout: 5243134471 lvb_type: 0 Sep 14 14:00:41 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.41@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Sep 14 14:02:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 36d53d1e-e3eb-2a5a-86b1-275956e5698b (at 10.8.8.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567bb400, cur 1536958955 expire 1536958805 last 1536958728 Sep 14 14:02:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 14:06:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 156573be-eca5-3a48-3b0f-a12ef5bbd1ab (at 10.9.102.26@o2ib4) Sep 14 14:06:19 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 14:07:44 oak-md1-s2 kernel: LustreError: 4304:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.65@o2ib4) returned error from blocking AST (req@ffff881c5bb89500 x1610627180628400 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881a8ba28a00/0x6c00c3e6a6a63fdc lrc: 4/0,0 mode: PR/PR res: [0x20001621e:0x1832a:0x0].0x0 bits 0x13 rrc: 40 type: IBT flags: 0x60200400000020 nid: 10.9.105.65@o2ib4 remote: 0xdca80cb3bee28c26 expref: 137 pid: 10671 timeout: 5243556989 lvb_type: 0 Sep 14 14:07:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 156573be-eca5-3a48-3b0f-a12ef5bbd1ab (at 10.9.102.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882005c37c00, cur 1536959264 expire 1536959114 last 1536959037 Sep 14 14:07:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 14:07:44 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.65@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Sep 14 14:18:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c21fa2df-da94-88ce-d0ba-f54693add6ea (at 10.8.12.2@o2ib6) Sep 14 14:18:27 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Sep 14 14:27:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b76ea0a-8a2e-0b62-84dd-0e128616b2f7 (at 10.8.21.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e72c57800, cur 1536960478 expire 1536960328 last 1536960251 Sep 14 14:27:58 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 14:28:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e7fec498-8db3-3e88-abdc-898065f3defa (at 10.9.106.55@o2ib4) Sep 14 14:28:54 oak-md1-s2 kernel: Lustre: Skipped 20 previous similar messages Sep 14 14:29:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e340b60d-b9ac-4433-fc36-4dc3e047f998 (at 10.8.12.32@o2ib6) in 224 seconds. I think it's dead, and I am evicting it. exp ffff8800567b8800, cur 1536960554 expire 1536960404 last 1536960330 Sep 14 14:32:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f670850-d052-cbf0-fd4b-62aa8eb5de75 (at 10.8.11.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102dfeb000, cur 1536960724 expire 1536960574 last 1536960497 Sep 14 14:32:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 14:37:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c2d26e2-ef20-43fc-5e12-96762bcb1f68 (at 10.8.13.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881038283c00, cur 1536961033 expire 1536960883 last 1536960806 Sep 14 14:39:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c86a420f-c781-b7d5-4291-9e94489b056b (at 10.8.2.16@o2ib6) Sep 14 14:39:07 oak-md1-s2 kernel: Lustre: Skipped 30 previous similar messages Sep 14 14:45:28 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 14:49:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 44d3c723-31a9-c609-7971-480bd39af9f1 (at 10.8.25.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802c8b66800, cur 1536961748 expire 1536961598 last 1536961521 Sep 14 14:49:08 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 14:53:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e340b60d-b9ac-4433-fc36-4dc3e047f998 (at 10.8.12.32@o2ib6) Sep 14 14:53:08 oak-md1-s2 kernel: Lustre: Skipped 19 previous similar messages Sep 14 15:01:58 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 15:03:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c7402119-795b-2902-b485-faa64d2953aa (at 10.8.21.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8820098dbc00, cur 1536962603 expire 1536962453 last 1536962376 Sep 14 15:03:23 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 15:07:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7e53f575-645c-b283-f09e-81226f104e0b (at 10.8.25.6@o2ib6) Sep 14 15:07:14 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 15:13:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc98793a-82e5-b8e4-863f-584fce147d29 (at 10.9.104.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b63d25c00, cur 1536963228 expire 1536963078 last 1536963001 Sep 14 15:13:48 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 15:18:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f41fcf7c-44d1-6c82-6057-02cf3b2d4405 (at 10.8.2.21@o2ib6) Sep 14 15:18:58 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 15:28:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f46d10b9-2fbb-6ff8-9c28-4bb5123ca47a (at 10.8.8.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a01f000, cur 1536964095 expire 1536963945 last 1536963868 Sep 14 15:28:15 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 14 15:30:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85d90c94-e070-68cc-c88b-2b519e41b039 (at 10.8.30.36@o2ib6) Sep 14 15:30:07 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 15:40:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 203b773a-81c3-b207-3bf4-bf1c8a54f05d (at 10.8.23.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880970fba400, cur 1536964802 expire 1536964652 last 1536964575 Sep 14 15:40:02 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 15:45:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bccf629c-900e-c4a9-171a-d70129138265 (at 10.8.14.1@o2ib6) Sep 14 15:45:34 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Sep 14 15:51:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7159cdbf-1fb2-6998-edc1-458a2962bb81 (at 10.8.12.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567b9000, cur 1536965495 expire 1536965345 last 1536965268 Sep 14 15:51:35 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 15:56:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbb9db85-60f5-adbd-5b0f-c1781850eda7 (at 10.8.19.8@o2ib6) Sep 14 15:56:33 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 16:01:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d796643a-0202-3637-62f3-97ef3a48e7e7 (at 10.9.106.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f646000, cur 1536966102 expire 1536965952 last 1536965875 Sep 14 16:01:42 oak-md1-s2 kernel: Lustre: Skipped 35 previous similar messages Sep 14 16:10:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 20156fdd-9270-a35c-4b98-63607ba2969e (at 10.9.107.2@o2ib4) Sep 14 16:10:19 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 16:12:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0b34220-337c-17c8-4974-47b3253ddcce (at 10.9.102.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200d937000, cur 1536966775 expire 1536966625 last 1536966548 Sep 14 16:12:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 16:20:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to af988c0b-7273-542c-11c0-7de39b50f62e (at 10.8.18.10@o2ib6) Sep 14 16:20:59 oak-md1-s2 kernel: Lustre: Skipped 30 previous similar messages Sep 14 16:21:38 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 16:21:38 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 14 16:23:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d403365-ced9-c759-9ac9-78bf3cb4a4a2 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880109889000, cur 1536967435 expire 1536967285 last 1536967208 Sep 14 16:23:55 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 16:31:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 93746428-2f58-7d78-871d-b7b2cb66dc27 (at 10.8.18.20@o2ib6) Sep 14 16:31:59 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 16:35:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cd694979-6094-08f6-4b8a-145cc3008499 (at 10.9.107.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3b9400, cur 1536968143 expire 1536967993 last 1536967916 Sep 14 16:35:43 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 16:42:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 14 16:42:55 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 16:52:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cc9bc5b8-76f0-d5d2-6a45-6cca2c57e573 (at 10.9.102.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e5e000, cur 1536969138 expire 1536968988 last 1536968911 Sep 14 16:52:18 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 16:53:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 004c122b-02e7-6a9f-39c6-78051b0a1809 (at 10.8.26.27@o2ib6) Sep 14 16:53:39 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 17:04:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ce512e0e-3a0f-c89d-e7a4-a8b5284798e3 (at 10.9.104.49@o2ib4) Sep 14 17:04:45 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 17:05:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 26d94f91-ec03-cf64-f96e-5a8df8f62165 (at 10.9.104.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e3a3800, cur 1536969937 expire 1536969787 last 1536969710 Sep 14 17:05:37 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 17:18:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b48c640-0817-52ca-42bd-3ac916481a6f (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880456224800, cur 1536970732 expire 1536970582 last 1536970505 Sep 14 17:18:52 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Sep 14 17:18:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fd8236e-050b-c62d-2664-f6711708a9ca (at 10.8.29.6@o2ib6) Sep 14 17:18:54 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 17:31:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Sep 14 17:31:37 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Sep 14 17:33:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7a9156fb-8791-b2dd-fc1e-6f9c89cb8841 (at 10.8.1.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103890a000, cur 1536971622 expire 1536971472 last 1536971395 Sep 14 17:43:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7a9156fb-8791-b2dd-fc1e-6f9c89cb8841 (at 10.8.1.17@o2ib6) Sep 14 17:43:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 17:48:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a3711652-2225-f403-251a-9e82e770218d (at 10.8.20.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2c9000, cur 1536972518 expire 1536972368 last 1536972291 Sep 14 17:48:38 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 17:58:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a252fbd9-2107-63c6-1aa8-6578bcb83073 (at 10.8.10.35@o2ib6) in 193 seconds. I think it's dead, and I am evicting it. exp ffff880963365000, cur 1536973125 expire 1536972975 last 1536972932 Sep 14 17:58:45 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 17:59:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 070273c2-258c-4544-ed92-0fac1849eede (at 10.9.102.56@o2ib4) Sep 14 17:59:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 18:12:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e58be353-6bc0-1c69-b54d-1db7c01f7d82 (at 10.9.112.17@o2ib4) Sep 14 18:12:56 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 18:13:07 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536973980/real 1536973980] req@ffff881bf8e9bf00 x1610627198244688/t0(0) o104->oak-MDT0000@10.9.105.32@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536973987 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 14 18:13:07 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 14 18:13:14 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536973987/real 1536973987] req@ffff881bf8e9bf00 x1610627198244688/t0(0) o104->oak-MDT0000@10.9.105.32@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536973994 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 18:13:28 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536974001/real 1536974001] req@ffff881bf8e9bf00 x1610627198244688/t0(0) o104->oak-MDT0000@10.9.105.32@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536974008 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 18:13:28 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 14 18:13:49 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536974022/real 1536974022] req@ffff881bf8e9bf00 x1610627198244688/t0(0) o104->oak-MDT0000@10.9.105.32@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536974029 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 18:13:49 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 18:14:24 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536974057/real 1536974057] req@ffff881bf8e9bf00 x1610627198244688/t0(0) o104->oak-MDT0000@10.9.105.32@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536974064 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 18:14:24 oak-md1-s2 kernel: Lustre: 10646:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 14 18:14:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 063c5f1d-526f-ca26-1ccb-95381185e608 (at 10.9.105.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b64ec00, cur 1536974084 expire 1536973934 last 1536973857 Sep 14 18:14:44 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 18:14:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 06717f7d-e3d0-b41d-98d4-62cae162f633 (at 10.9.113.3@o2ib4) reconnecting Sep 14 18:14:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 18:14:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7edfc4e1-499e-8b03-a014-f9e032f4d537 (at 10.8.28.1@o2ib6) reconnecting Sep 14 18:15:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 06717f7d-e3d0-b41d-98d4-62cae162f633 (at 10.9.113.3@o2ib4) reconnecting Sep 14 18:22:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a252fbd9-2107-63c6-1aa8-6578bcb83073 (at 10.8.10.35@o2ib6) Sep 14 18:22:59 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 18:34:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 063c5f1d-526f-ca26-1ccb-95381185e608 (at 10.9.105.72@o2ib4) Sep 14 18:34:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 18:34:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9b0c722-5ebb-2c73-cef1-db3658bd947a (at 10.8.24.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063426800, cur 1536975277 expire 1536975127 last 1536975050 Sep 14 18:34:37 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 18:47:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 951c9ac6-b471-92e7-7d84-70391f29b3f2 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e19c43c00, cur 1536976066 expire 1536975916 last 1536975839 Sep 14 18:47:46 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 14 18:47:54 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 18:50:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6c260e25-84e3-f0df-7485-52b1ed7d2cd1 (at 10.8.29.5@o2ib6) Sep 14 18:50:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 18:58:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 23cf492c-730a-7bf7-a7e3-af1d811b9767 (at 10.9.112.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063421000, cur 1536976694 expire 1536976544 last 1536976467 Sep 14 18:58:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 19:03:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9f20104c-ec76-03cb-654f-2a5484a6721e (at 10.8.26.34@o2ib6) Sep 14 19:03:17 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 14 19:09:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 57d93be5-1603-d8ba-e857-f0b78ec56438 (at 10.8.19.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e6ba0b400, cur 1536977385 expire 1536977235 last 1536977158 Sep 14 19:09:45 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 19:13:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 23516c40-6752-1a5d-abee-85000b467993 (at 10.8.13.2@o2ib6) Sep 14 19:13:42 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 14 19:24:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a51c2751-2210-5632-0d46-a3c2ad9d9c6e (at 10.8.22.33@o2ib6) Sep 14 19:24:39 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 19:26:52 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 14 19:32:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4f4727a-52fd-6426-ef11-572959991e3f (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c870c00, cur 1536978738 expire 1536978588 last 1536978511 Sep 14 19:32:18 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 19:33:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cc19ebe2-50de-dba1-3657-0406375fb797 (at 10.8.18.13@o2ib6) in 200 seconds. I think it's dead, and I am evicting it. exp ffff880ea36fe400, cur 1536978814 expire 1536978664 last 1536978614 Sep 14 19:33:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 19:37:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0a2db5a-a04f-2beb-7817-bdc136c7d38c (at 10.9.104.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef28f6800, cur 1536979045 expire 1536978895 last 1536978818 Sep 14 19:37:25 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 19:42:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f28d8bfc-6c86-b70e-0a23-c5732fa66f3d (at 10.8.18.13@o2ib6) Sep 14 19:42:02 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 19:43:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client abfe1850-8d93-1bc2-454d-b1d26ce9a537 (at 10.8.17.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567bfc00, cur 1536979394 expire 1536979244 last 1536979167 Sep 14 19:43:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 19:52:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 14 19:52:09 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 20:02:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe2c119c-25cd-87cc-82bb-6d149f1e7278 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c40fe6800, cur 1536980556 expire 1536980406 last 1536980329 Sep 14 20:02:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 20:02:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f6b8524e-5d1c-5581-c3f3-1b670faa6f0a (at 10.9.102.2@o2ib4) Sep 14 20:02:44 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 20:07:46 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536980859/real 1536980859] req@ffff880d0687dd00 x1610627205360032/t0(0) o104->oak-MDT0000@10.9.112.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536980866 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 14 20:07:46 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 20:08:01 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536980873/real 1536980873] req@ffff880d0687dd00 x1610627205360032/t0(0) o104->oak-MDT0000@10.9.112.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536980880 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 20:08:01 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 14 20:08:22 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536980895/real 1536980895] req@ffff880d0687dd00 x1610627205360032/t0(0) o104->oak-MDT0000@10.9.112.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536980902 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 20:08:22 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 20:08:57 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536980930/real 1536980930] req@ffff880d0687dd00 x1610627205360032/t0(0) o104->oak-MDT0000@10.9.112.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536980937 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 20:08:57 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 14 20:10:07 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536981000/real 1536981000] req@ffff880d0687dd00 x1610627205360032/t0(0) o104->oak-MDT0000@10.9.112.6@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536981007 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 20:10:07 oak-md1-s2 kernel: Lustre: 10517:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Sep 14 20:11:00 oak-md1-s2 kernel: LNet: Service thread pid 10517 was inactive for 200.58s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 14 20:11:00 oak-md1-s2 kernel: LNet: Skipped 2 previous similar messages Sep 14 20:11:00 oak-md1-s2 kernel: Pid: 10517, comm: mdt00_040 Sep 14 20:11:00 oak-md1-s2 kernel: Sep 14 20:11:00 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 14 20:11:00 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 14 20:11:00 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 14 20:11:00 oak-md1-s2 kernel: [] ptlrpc_set_wait+0x4c0/0x910 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 14 20:11:00 oak-md1-s2 kernel: [] ldlm_run_ast_work+0xd3/0x3a0 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ldlm_handle_conflict_lock+0x75/0x330 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ldlm_process_inodebits_lock+0xfd/0x400 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x42e/0x970 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? ldlm_lock_create+0x1fc/0xa30 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x1c3/0x860 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 14 20:11:00 oak-md1-s2 kernel: [] mdt_object_local_lock+0x452/0xaf0 [mdt] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? lu_object_find_at+0x95/0x290 [obdclass] Sep 14 20:11:00 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 14 20:11:00 oak-md1-s2 kernel: [] mdt_object_lock+0x14/0x20 [mdt] Sep 14 20:11:00 oak-md1-s2 kernel: [] mdt_create+0x419/0xbb0 [mdt] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? lprocfs_counter_add+0xf9/0x160 [obdclass] Sep 14 20:11:00 oak-md1-s2 kernel: [] ? lprocfs_job_stats_log+0xd1/0x630 [obdclass] Sep 14 20:11:00 oak-md1-s2 kernel: [] mdt_reint_create+0x16b/0x350 [mdt] Sep 14 20:11:00 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 14 20:11:01 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 14 20:11:01 oak-md1-s2 kernel: [] mdt_reint+0x67/0x140 [mdt] Sep 14 20:11:01 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 14 20:11:01 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 14 20:11:01 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 14 20:11:01 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 14 20:11:01 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 14 20:11:01 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 14 20:11:01 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 14 20:11:01 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 14 20:11:01 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 14 20:11:01 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 14 20:11:01 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 14 20:11:01 oak-md1-s2 kernel: Sep 14 20:11:01 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1536981061.10517 Sep 14 20:11:02 oak-md1-s2 kernel: LNet: Service thread pid 10517 completed after 202.22s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Sep 14 20:11:02 oak-md1-s2 kernel: LNet: Skipped 3 previous similar messages Sep 14 20:14:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b370ec9f-692c-6b23-9a3e-c7664a8455ae (at 10.8.22.11@o2ib6) Sep 14 20:14:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 20:19:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5bece9f-1994-cc1e-a006-b95a2b0d4482 (at 10.9.104.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8698c00, cur 1536981582 expire 1536981432 last 1536981355 Sep 14 20:19:42 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 20:25:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0f8c754a-5656-24d4-e0d6-f97c9c90abbb (at 10.9.102.16@o2ib4) Sep 14 20:25:21 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 20:30:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 095a6506-2edd-aa30-01cf-ac1b665245a5 (at 10.8.10.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880814646400, cur 1536982255 expire 1536982105 last 1536982028 Sep 14 20:30:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 20:41:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b5bece9f-1994-cc1e-a006-b95a2b0d4482 (at 10.9.104.1@o2ib4) Sep 14 20:41:25 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 20:43:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e69305dd-c324-76b2-02a0-2ddef3efb42b (at 10.8.29.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567ba800, cur 1536983022 expire 1536982872 last 1536982795 Sep 14 20:43:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 20:54:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c9099b4e-3665-e155-4108-e5278e5d37c7 (at 10.9.104.62@o2ib4) Sep 14 20:54:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 20:55:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e054ea7f-9af6-6009-eb9a-5c9eddc957f4 (at 10.9.104.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef28f1000, cur 1536983712 expire 1536983562 last 1536983485 Sep 14 20:55:12 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 21:03:00 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536984173/real 1536984173] req@ffff881c0abbb000 x1610627208673072/t0(0) o104->oak-MDT0000@10.9.104.35@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536984180 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 14 21:03:00 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Sep 14 21:03:21 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536984194/real 1536984194] req@ffff881c0abbb000 x1610627208673072/t0(0) o104->oak-MDT0000@10.9.104.35@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536984201 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 21:03:21 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 21:03:56 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536984229/real 1536984229] req@ffff881c0abbb000 x1610627208673072/t0(0) o104->oak-MDT0000@10.9.104.35@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536984236 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 21:03:56 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 14 21:05:06 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536984299/real 1536984299] req@ffff881c0abbb000 x1610627208673072/t0(0) o104->oak-MDT0000@10.9.104.35@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536984306 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 21:05:06 oak-md1-s2 kernel: Lustre: 10613:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Sep 14 21:05:28 oak-md1-s2 kernel: LustreError: 10613:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.104.35@o2ib4) failed to reply to blocking AST (req@ffff881c0abbb000 x1610627208673072 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880058258800/0x6c00c3e79330dc90 lrc: 4/0,0 mode: PR/PR res: [0x200017482:0x18d7c:0x0].0x0 bits 0x13 rrc: 25 type: IBT flags: 0x60200400000020 nid: 10.9.104.35@o2ib4 remote: 0x5f96520833be1345 expref: 130 pid: 10566 timeout: 5268663926 lvb_type: 0 Sep 14 21:05:28 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.104.35@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Sep 14 21:06:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 119f7486-6b19-c459-64bb-c376586d7bf0 (at 10.8.24.22@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff88101bf11000, cur 1536984363 expire 1536984213 last 1536984192 Sep 14 21:06:03 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 21:06:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f4ab7c-96d6-a80d-e451-2fe219bc7544 (at 10.8.30.10@o2ib6) Sep 14 21:06:48 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 21:16:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e054ea7f-9af6-6009-eb9a-5c9eddc957f4 (at 10.9.104.67@o2ib4) Sep 14 21:16:58 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 21:19:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9735ccc7-8259-ff1c-fd85-909e0c4749e6 (at 10.9.114.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882038608000, cur 1536985140 expire 1536984990 last 1536984913 Sep 14 21:19:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 21:27:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d864271d-68ba-37ce-9609-2c6659593006 (at 10.9.105.28@o2ib4) Sep 14 21:27:20 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 21:29:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e90ff330-3085-7576-cce2-638790dc16d3 (at 10.8.23.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fe22c00, cur 1536985761 expire 1536985611 last 1536985534 Sep 14 21:29:21 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 21:31:53 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536985906/real 1536985906] req@ffff88168c67a400 x1610627210341936/t0(0) o104->oak-MDT0000@10.9.105.60@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536985913 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 14 21:31:53 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Sep 14 21:32:14 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536985927/real 1536985927] req@ffff88168c67a400 x1610627210341936/t0(0) o104->oak-MDT0000@10.9.105.60@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536985934 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 21:32:14 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 21:32:49 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536985962/real 1536985962] req@ffff88168c67a400 x1610627210341936/t0(0) o104->oak-MDT0000@10.9.105.60@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536985969 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 21:32:49 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 14 21:33:59 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536986032/real 1536986032] req@ffff88168c67a400 x1610627210341936/t0(0) o104->oak-MDT0000@10.9.105.60@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536986039 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 21:33:59 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Sep 14 21:34:20 oak-md1-s2 kernel: LustreError: 10659:0:(service.c:2077:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.8.15.2@o2ib6: deadline 100:37s ago Sep 14 21:34:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 50b5e033-e287-5c20-b2f0-b8f9cf4514a4 (at 10.8.15.2@o2ib6) reconnecting Sep 14 21:34:20 oak-md1-s2 kernel: Lustre: 10659:0:(service.c:2114:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (100:37s); client may timeout. req@ffff8802fda4b000 x1611646550147488/t0(0) o38->@:0/0 lens 520/0 e 0 to 0 dl 1536986023 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Sep 14 21:34:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 50b5e033-e287-5c20-b2f0-b8f9cf4514a4 (at 10.8.15.2@o2ib6) reconnecting Sep 14 21:34:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 21:34:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 94a51969-8bf2-aecc-0815-c33c5fc6774a (at 10.9.114.14@o2ib4) reconnecting Sep 14 21:38:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 24c3fca1-67d3-2aba-607e-e27143a5de35 (at 10.8.12.14@o2ib6) Sep 14 21:38:39 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 14 21:40:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b013eda-077a-097a-0fba-6687e4b3366b (at 10.9.106.18@o2ib4) in 282 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3be800, cur 1536986446 expire 1536986296 last 1536986164 Sep 14 21:40:46 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 14 21:49:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 90601611-f7cd-ad01-4385-1cd22e98279c (at 10.8.8.31@o2ib6) Sep 14 21:49:19 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 21:50:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a580058c-5115-a238-9053-b324366eddc1 (at 10.9.106.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a3e6000, cur 1536987051 expire 1536986901 last 1536986824 Sep 14 21:50:51 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 22:01:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2ce6fa59-5e31-71ae-af70-fd0bf18fb38d (at 10.9.104.4@o2ib4) Sep 14 22:01:15 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Sep 14 22:02:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ce6fa59-5e31-71ae-af70-fd0bf18fb38d (at 10.9.104.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882006162c00, cur 1536987721 expire 1536987571 last 1536987494 Sep 14 22:02:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 22:13:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ad32a28-c1bb-d588-df17-32e18e07516c (at 10.8.18.12@o2ib6) Sep 14 22:13:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 22:14:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f846d569-cfa3-2fc9-94d7-4df6a78b42ef (at 10.9.104.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b744400, cur 1536988488 expire 1536988338 last 1536988261 Sep 14 22:14:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 22:22:46 oak-md1-s2 kernel: Lustre: 10590:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536988959/real 1536988959] req@ffff880bba194800 x1610627213399696/t0(0) o104->oak-MDT0000@10.9.105.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536988966 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 14 22:22:46 oak-md1-s2 kernel: Lustre: 10590:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 22:23:07 oak-md1-s2 kernel: Lustre: 10590:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536988980/real 1536988980] req@ffff880bba194800 x1610627213399696/t0(0) o104->oak-MDT0000@10.9.105.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536988987 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 22:23:07 oak-md1-s2 kernel: Lustre: 10590:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 14 22:23:42 oak-md1-s2 kernel: Lustre: 10590:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536989015/real 1536989015] req@ffff880bba194800 x1610627213399696/t0(0) o104->oak-MDT0000@10.9.105.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536989022 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 14 22:23:42 oak-md1-s2 kernel: Lustre: 10590:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 14 22:24:25 oak-md1-s2 kernel: LustreError: 10590:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.62@o2ib4) failed to reply to blocking AST (req@ffff880bba194800 x1610627213399696 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880a4c033400/0x6c00c3e7a91880ca lrc: 4/0,0 mode: PR/PR res: [0x20001621e:0x1832a:0x0].0x0 bits 0x13 rrc: 490 type: IBT flags: 0x60200400000020 nid: 10.9.105.62@o2ib4 remote: 0xd64901c900bd2536 expref: 137 pid: 10507 timeout: 5273350913 lvb_type: 0 Sep 14 22:24:25 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.62@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Sep 14 22:24:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 86e301e8-a840-cff3-1355-933f8c7263d9 (at 10.8.24.26@o2ib6) Sep 14 22:24:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Export ffff8807fa9a9400 already connecting from 10.8.24.26@o2ib6 Sep 14 22:24:25 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 22:24:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 590c101f-9072-5fe4-67f6-9ef5670ade95 (at 10.8.24.26@o2ib6) reconnecting Sep 14 22:26:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b646516-f96c-d11e-6efd-2c538127e0d7 (at 10.8.8.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe5a5d800, cur 1536989209 expire 1536989059 last 1536988982 Sep 14 22:26:49 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 14 22:37:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c94d08be-0c05-348d-22c7-f38336d0d93a (at 10.9.104.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b64d000, cur 1536989840 expire 1536989690 last 1536989613 Sep 14 22:37:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 22:38:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 91d32932-55c9-4ae2-802e-011dbc943b30 (at 10.9.102.37@o2ib4) Sep 14 22:38:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 22:50:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b646516-f96c-d11e-6efd-2c538127e0d7 (at 10.8.8.11@o2ib6) Sep 14 22:50:18 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 22:50:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b1d42fe-33d6-90fe-73b3-710e58c0634b (at 10.8.28.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd7a7e800, cur 1536990646 expire 1536990496 last 1536990419 Sep 14 22:50:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 23:02:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 86a8a040-246f-39c3-c30b-de653488110b (at 10.9.101.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c631000, cur 1536991346 expire 1536991196 last 1536991119 Sep 14 23:02:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 23:09:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7b1d42fe-33d6-90fe-73b3-710e58c0634b (at 10.8.28.10@o2ib6) Sep 14 23:09:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 23:14:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55d74724-b2a5-19c3-b749-ab2658f18863 (at 10.9.101.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca4cc00, cur 1536992083 expire 1536991933 last 1536991856 Sep 14 23:14:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 23:20:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 41ccc003-fd4b-2f86-caca-34533260eba6 (at 10.8.10.24@o2ib6) Sep 14 23:20:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 14 23:32:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6835a312-15ce-ce5b-9f12-9c85c224260d (at 10.9.106.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc4cc3400, cur 1536993153 expire 1536993003 last 1536992926 Sep 14 23:32:33 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 14 23:35:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 55d74724-b2a5-19c3-b749-ab2658f18863 (at 10.9.101.43@o2ib4) Sep 14 23:35:10 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 14 23:46:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a0cbb64-73cf-8a7a-8fbb-52e091912fb7 (at 10.8.11.36@o2ib6) Sep 14 23:46:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 14 23:50:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dbc641a0-13b1-ee28-c241-c0c40b99b57e (at 10.8.17.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a106a7400, cur 1536994255 expire 1536994105 last 1536994028 Sep 14 23:50:55 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 00:00:04 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 00:00:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0d7e2f10-55e7-eba9-60cf-7542560f0687 (at 10.8.23.19@o2ib6) Sep 15 00:00:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 00:01:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 03291c92-bab1-fcc6-67fb-df43e82d4ade (at 10.8.26.20@o2ib6) in 223 seconds. I think it's dead, and I am evicting it. exp ffff880b63d23000, cur 1536994903 expire 1536994753 last 1536994680 Sep 15 00:01:43 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 00:10:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4bd39c9b-b37b-0510-e395-291cf356c46e (at 10.8.27.24@o2ib6) Sep 15 00:10:10 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 00:10:56 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536995449/real 1536995449] req@ffff881860bf8300 x1610627221680512/t0(0) o104->oak-MDT0000@10.9.104.34@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536995456 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 15 00:10:56 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Sep 15 00:11:11 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536995463/real 1536995463] req@ffff881860bf8300 x1610627221680512/t0(0) o104->oak-MDT0000@10.9.104.34@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536995470 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 00:11:11 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 15 00:11:32 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536995485/real 1536995485] req@ffff881860bf8300 x1610627221680512/t0(0) o104->oak-MDT0000@10.9.104.34@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536995492 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 00:11:32 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 15 00:12:07 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536995520/real 1536995520] req@ffff881860bf8300 x1610627221680512/t0(0) o104->oak-MDT0000@10.9.104.34@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536995527 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 00:12:07 oak-md1-s2 kernel: Lustre: 10609:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Sep 15 00:13:11 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536995584/real 1536995584] req@ffff880fa8d94b00 x1610627221723456/t0(0) o104->oak-MDT0000@10.9.104.34@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536995591 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 00:13:11 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Sep 15 00:13:24 oak-md1-s2 kernel: LustreError: 10609:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.104.34@o2ib4) failed to reply to blocking AST (req@ffff881860bf8300 x1610627221680512 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880042af4a00/0x6c00c3e7c96a582f lrc: 4/0,0 mode: CR/CR res: [0x200017488:0x35d:0x0].0x0 bits 0x9 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.104.34@o2ib4 remote: 0xfceed0118394c0e5 expref: 256 pid: 10485 timeout: 5279940345 lvb_type: 0 Sep 15 00:13:24 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.104.34@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Sep 15 00:13:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 059d17f6-cb26-4ddb-f51f-6839df6123c1 (at 10.9.104.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef2887000, cur 1536995610 expire 1536995460 last 1536995383 Sep 15 00:13:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 00:16:00 oak-md1-s2 kernel: Lustre: 10502:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536995753/real 1536995753] req@ffff881a78613c00 x1610627222069072/t0(0) o104->oak-MDT0000@10.9.105.48@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536995760 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 15 00:16:00 oak-md1-s2 kernel: Lustre: 10502:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Sep 15 00:16:49 oak-md1-s2 kernel: LustreError: 10502:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.48@o2ib4) returned error from blocking AST (req@ffff881a78613c00 x1610627222069072 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880ac2b25c00/0x6c00c3e7c941b962 lrc: 4/0,0 mode: PR/PR res: [0x20001621e:0x1832a:0x0].0x0 bits 0x13 rrc: 470 type: IBT flags: 0x60200400000020 nid: 10.9.105.48@o2ib4 remote: 0xaa187af88488828d expref: 132 pid: 10547 timeout: 5280102714 lvb_type: 0 Sep 15 00:16:49 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.48@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Sep 15 00:21:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4f479a3c-206c-7d80-24a4-745dd7f49f2e (at 10.9.104.69@o2ib4) Sep 15 00:21:45 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 00:23:13 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 00:23:13 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 15 00:24:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2bcffeb-24ac-3558-3eca-f02626d5362e (at 10.8.8.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092ea06c00, cur 1536996281 expire 1536996131 last 1536996054 Sep 15 00:24:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 00:34:54 oak-md1-s2 kernel: Lustre: 4679:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536996887/real 1536996887] req@ffff881c70363000 x1610627224049392/t0(0) o104->oak-MDT0000@10.9.105.61@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536996894 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 15 00:34:54 oak-md1-s2 kernel: Lustre: 4679:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Sep 15 00:34:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7194205-b2c9-bce5-f35b-d3d4bc5723ab (at 10.8.8.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882002f78400, cur 1536996896 expire 1536996746 last 1536996669 Sep 15 00:34:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 00:35:30 oak-md1-s2 kernel: Lustre: 4679:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536996922/real 1536996922] req@ffff881c70363000 x1610627224049392/t0(0) o104->oak-MDT0000@10.9.105.61@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536996929 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 00:35:30 oak-md1-s2 kernel: Lustre: 4679:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 15 00:36:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 059d17f6-cb26-4ddb-f51f-6839df6123c1 (at 10.9.104.20@o2ib4) Sep 15 00:36:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 00:36:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 23c47258-c83c-dc25-951c-8490adee9b34 (at 10.9.104.20@o2ib4) reconnecting Sep 15 00:45:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e282a2c1-e19b-54e8-7a33-bbb74a09b2a4 (at 10.8.17.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001dd0800, cur 1536997535 expire 1536997385 last 1536997308 Sep 15 00:45:35 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 00:49:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a2bcffeb-24ac-3558-3eca-f02626d5362e (at 10.8.8.9@o2ib6) Sep 15 00:49:43 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 01:03:28 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536998600/real 1536998600] req@ffff881a75384e00 x1610627227205776/t0(0) o104->oak-MDT0000@10.9.105.42@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536998607 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 15 01:03:28 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Sep 15 01:03:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b57bffd-9dc4-b1cb-17fd-e78b7b2589f8 (at 10.8.31.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096bf07800, cur 1536998620 expire 1536998470 last 1536998393 Sep 15 01:03:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 01:03:42 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536998615/real 1536998615] req@ffff881a75384e00 x1610627227205776/t0(0) o104->oak-MDT0000@10.9.105.42@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536998622 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 01:03:42 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 15 01:04:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e282a2c1-e19b-54e8-7a33-bbb74a09b2a4 (at 10.8.17.16@o2ib6) Sep 15 01:04:00 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 01:04:03 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536998636/real 1536998636] req@ffff881a75384e00 x1610627227205776/t0(0) o104->oak-MDT0000@10.9.105.42@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536998643 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 01:04:03 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 15 01:04:38 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536998671/real 1536998671] req@ffff881a75384e00 x1610627227205776/t0(0) o104->oak-MDT0000@10.9.105.42@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536998678 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 01:04:38 oak-md1-s2 kernel: Lustre: 10670:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 15 01:05:06 oak-md1-s2 kernel: LustreError: 10670:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.42@o2ib4) failed to reply to blocking AST (req@ffff881a75384e00 x1610627227205776 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881202f6b400/0x6c00c3e7d72b4f14 lrc: 4/0,0 mode: PR/PR res: [0x20001621e:0x1832a:0x0].0x0 bits 0x13 rrc: 464 type: IBT flags: 0x60200400000020 nid: 10.9.105.42@o2ib4 remote: 0x637cf1c65f4f8f15 expref: 123 pid: 10615 timeout: 5282992480 lvb_type: 0 Sep 15 01:05:06 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.42@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Sep 15 01:15:17 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 01:15:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Sep 15 01:15:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 01:16:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8de3f372-d077-3849-ea61-a674d74fdadc (at 10.8.23.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe444dc00, cur 1536999370 expire 1536999220 last 1536999143 Sep 15 01:16:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 01:26:19 oak-md1-s2 kernel: Lustre: 10673:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536999972/real 1536999972] req@ffff881ff0b1b600 x1610627228811584/t0(0) o104->oak-MDT0000@10.9.105.64@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536999979 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 15 01:26:19 oak-md1-s2 kernel: Lustre: 10673:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Sep 15 01:26:33 oak-md1-s2 kernel: Lustre: 10673:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1536999986/real 1536999986] req@ffff881ff0b1b600 x1610627228811584/t0(0) o104->oak-MDT0000@10.9.105.64@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1536999993 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 01:26:33 oak-md1-s2 kernel: Lustre: 10673:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 15 01:26:54 oak-md1-s2 kernel: Lustre: 10673:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537000007/real 1537000007] req@ffff881ff0b1b600 x1610627228811584/t0(0) o104->oak-MDT0000@10.9.105.64@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1537000014 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 15 01:26:54 oak-md1-s2 kernel: Lustre: 10673:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Sep 15 01:27:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fd0cf0ae-191f-9958-55c1-2eaf3bb58918 (at 10.9.105.64@o2ib4) Sep 15 01:27:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 01:27:16 oak-md1-s2 kernel: LustreError: 10673:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.64@o2ib4) returned error from blocking AST (req@ffff881ff0b1b600 x1610627228811584 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8811b37a0800/0x6c00c3e7dcc47f93 lrc: 4/0,0 mode: PR/PR res: [0x20001621e:0x1832a:0x0].0x0 bits 0x13 rrc: 437 type: IBT flags: 0x60200400000020 nid: 10.9.105.64@o2ib4 remote: 0x80aa53e96e0f39af expref: 145 pid: 10539 timeout: 5284328952 lvb_type: 0 Sep 15 01:27:16 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.64@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Sep 15 01:39:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 76284cda-043c-3295-8d88-806c3bc3949d (at 10.8.22.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103890f400, cur 1537000773 expire 1537000623 last 1537000546 Sep 15 01:39:33 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 01:39:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8637d08e-c488-99b6-e567-295ff68f8921 (at 10.9.101.29@o2ib4) Sep 15 01:39:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 01:40:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8551ef9d-011c-3492-2971-544376b6e8ae (at 10.9.101.61@o2ib4) in 211 seconds. I think it's dead, and I am evicting it. exp ffff88019e614800, cur 1537000849 expire 1537000699 last 1537000638 Sep 15 01:48:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5461b3dd-b613-3b94-ee1b-e04ade1becd9 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ac9f46800, cur 1537001309 expire 1537001159 last 1537001082 Sep 15 01:58:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 84c724b9-b229-9a53-4296-09ee48f9e8fa (at 10.8.11.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2c9800, cur 1537001909 expire 1537001759 last 1537001682 Sep 15 02:00:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 76284cda-043c-3295-8d88-806c3bc3949d (at 10.8.22.30@o2ib6) Sep 15 02:00:00 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 02:06:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Sep 15 02:11:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ddff929a-cc60-fb27-9b8b-1a40efc890e0 (at 10.8.19.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2ccc00, cur 1537002670 expire 1537002520 last 1537002443 Sep 15 02:11:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 02:21:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 40d03a30-0722-73fb-2e85-6c7fa4fcb884 (at 10.9.101.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565cd000, cur 1537003315 expire 1537003165 last 1537003088 Sep 15 02:22:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 84c724b9-b229-9a53-4296-09ee48f9e8fa (at 10.8.11.34@o2ib6) Sep 15 02:22:03 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 02:24:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 71312e7c-31db-82d2-5b65-f87400966321 (at 10.9.105.39@o2ib4) Sep 15 02:27:41 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 02:39:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 463044d9-8fba-9e68-ba89-ce485724a8da (at 10.8.12.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe5a59800, cur 1537004386 expire 1537004236 last 1537004159 Sep 15 02:39:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 02:42:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 40d03a30-0722-73fb-2e85-6c7fa4fcb884 (at 10.9.101.12@o2ib4) Sep 15 02:43:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 79f0c6c7-0b2e-0258-fd29-427810f8977d (at 10.9.102.62@o2ib4) Sep 15 02:51:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 35db2d6b-44fe-b90e-6111-701958ebdb63 (at 10.8.24.24@o2ib6) Sep 15 02:57:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2219ba16-2ab9-6279-2dfa-83e04b252e69 (at 10.8.22.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe69db000, cur 1537005453 expire 1537005303 last 1537005226 Sep 15 02:57:33 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 15 03:01:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 91d3260c-64ae-ffe1-1260-3bb5c942217a (at 10.8.12.1@o2ib6) Sep 15 03:01:30 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 03:09:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 148072c7-5fae-0d3f-dc63-40a26012e148 (at 10.8.12.18@o2ib6) Sep 15 03:09:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 03:10:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f9ad3a4a-506b-662b-d78f-328ef43a18f9 (at 10.8.12.8@o2ib6) Sep 15 03:15:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2219ba16-2ab9-6279-2dfa-83e04b252e69 (at 10.8.22.2@o2ib6) Sep 15 03:15:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 03:26:07 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 03:31:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd7b6100-62ef-eb68-50d2-964aa8a16fa2 (at 10.9.106.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810391e0000, cur 1537007502 expire 1537007352 last 1537007275 Sep 15 03:38:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dec0eea-96a1-ba7d-453e-9beda7d0de53 (at 10.9.106.63@o2ib4) Sep 15 03:39:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5dec0eea-96a1-ba7d-453e-9beda7d0de53 (at 10.9.106.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a51c00, cur 1537007959 expire 1537007809 last 1537007732 Sep 15 03:42:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c30fd30-f45b-0199-e6f5-df773ec4f626 (at 10.9.107.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019ce1b000, cur 1537008144 expire 1537007994 last 1537007917 Sep 15 03:48:08 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 03:48:09 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 03:51:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3f35c8e8-585e-fd2c-7890-f7065a5794f5 (at 10.9.106.58@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a2f5400, cur 1537008666 expire 1537008516 last 1537008439 Sep 15 03:54:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd7b6100-62ef-eb68-50d2-964aa8a16fa2 (at 10.9.106.59@o2ib4) Sep 15 03:59:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3c30fd30-f45b-0199-e6f5-df773ec4f626 (at 10.9.107.3@o2ib4) Sep 15 04:04:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 740168ad-fda3-eb8f-8fa9-f92503b158d4 (at 10.9.102.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019ce1ec00, cur 1537009491 expire 1537009341 last 1537009264 Sep 15 04:04:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 04:13:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3f35c8e8-585e-fd2c-7890-f7065a5794f5 (at 10.9.106.58@o2ib4) Sep 15 04:14:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c6f744b5-7bf4-e9d9-ec06-0aead770d706 (at 10.8.29.7@o2ib6) Sep 15 04:21:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fa40c59-3477-9289-f07a-3a9c50b584d1 (at 10.9.105.50@o2ib4) Sep 15 04:21:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e27135d-365a-ab5f-a4dc-ed31b2e6fbfe (at 10.8.11.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca4ec00, cur 1537010496 expire 1537010346 last 1537010269 Sep 15 04:21:36 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 04:26:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 740168ad-fda3-eb8f-8fa9-f92503b158d4 (at 10.9.102.69@o2ib4) Sep 15 04:28:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a02d950-123b-9a4e-034a-363f1f622796 (at 10.9.106.54@o2ib4) Sep 15 04:28:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 04:30:14 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 04:30:14 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 15 04:31:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d8c9b12-b934-725d-aee3-9111c1723937 (at 10.8.30.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e6ba09c00, cur 1537011118 expire 1537010968 last 1537010891 Sep 15 04:44:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c189aa2f-5d15-2176-ef2a-dc8d4018f6c7 (at 10.9.106.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3be400, cur 1537011891 expire 1537011741 last 1537011664 Sep 15 04:44:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 04:45:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1e27135d-365a-ab5f-a4dc-ed31b2e6fbfe (at 10.8.11.3@o2ib6) Sep 15 04:51:54 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 04:51:54 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Sep 15 04:52:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d8c9b12-b934-725d-aee3-9111c1723937 (at 10.8.30.33@o2ib6) Sep 15 05:00:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e5ea46dc-4dfd-9ce5-49ca-f77bd0aaba15 (at 10.8.26.28@o2ib6) Sep 15 05:00:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2646041f-c477-2f9f-7756-78ce96c7b560 (at 10.9.106.64@o2ib4) Sep 15 05:07:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c189aa2f-5d15-2176-ef2a-dc8d4018f6c7 (at 10.9.106.57@o2ib4) Sep 15 05:08:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f386ef6e-504c-f223-0038-f702d016965e (at 10.9.105.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881850b04800, cur 1537013337 expire 1537013187 last 1537013110 Sep 15 05:10:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b99735f-d356-c688-7ee6-53ee18d9d991 (at 10.9.101.41@o2ib4) in 173 seconds. I think it's dead, and I am evicting it. exp ffff881dab42e800, cur 1537013413 expire 1537013263 last 1537013240 Sep 15 05:13:24 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 05:13:24 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 15 05:13:25 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 05:13:25 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 15 05:17:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dde24faf-89c1-368c-43c0-794cdb12fc97 (at 10.9.105.29@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e780400, cur 1537013832 expire 1537013682 last 1537013605 Sep 15 05:23:14 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 05:23:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a08e0b50-1685-981f-9335-883abb4e5bcc (at 10.9.105.34@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef287ac00, cur 1537014239 expire 1537014089 last 1537014012 Sep 15 05:23:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 05:30:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 87092071-edce-0207-d2f2-d4729884e50a (at 10.9.105.24@o2ib4) Sep 15 05:31:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3a815cbc-d650-5201-c8ac-6586d80e6809 (at 10.9.101.41@o2ib4) Sep 15 05:37:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7086ade7-a4ee-517c-8ef5-c963b55a575d (at 10.9.106.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103a9b8800, cur 1537015050 expire 1537014900 last 1537014823 Sep 15 05:38:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dde24faf-89c1-368c-43c0-794cdb12fc97 (at 10.9.105.29@o2ib4) Sep 15 05:43:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8679d4d2-bd86-69b2-4f23-d6e828cb4622 (at 10.9.104.71@o2ib4) Sep 15 05:45:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a08e0b50-1685-981f-9335-883abb4e5bcc (at 10.9.105.34@o2ib4) Sep 15 05:59:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c17c7504-f891-a009-27ca-0f6544f6e1e4 (at 10.9.114.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203860e800, cur 1537016373 expire 1537016223 last 1537016146 Sep 15 05:59:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 06:00:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7086ade7-a4ee-517c-8ef5-c963b55a575d (at 10.9.106.69@o2ib4) Sep 15 06:00:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 035f7e72-5f28-c5ed-1520-89b77f2bda76 (at 10.8.11.25@o2ib6) in 183 seconds. I think it's dead, and I am evicting it. exp ffff880b58f83000, cur 1537016449 expire 1537016299 last 1537016266 Sep 15 06:14:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c17c7504-f891-a009-27ca-0f6544f6e1e4 (at 10.9.114.13@o2ib4) Sep 15 06:16:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 034ffa36-ef99-d95c-9809-e828f35a85b6 (at 10.8.12.28@o2ib6) Sep 15 06:23:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75ec0917-15a4-7faf-a510-622c5966890d (at 10.8.10.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f63c400, cur 1537017785 expire 1537017635 last 1537017558 Sep 15 06:24:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7a29b7ea-27b6-3537-6761-540c92a0a76e (at 10.8.20.31@o2ib6) in 187 seconds. I think it's dead, and I am evicting it. exp ffff881fe721dc00, cur 1537017861 expire 1537017711 last 1537017674 Sep 15 06:24:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 06:25:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 035f7e72-5f28-c5ed-1520-89b77f2bda76 (at 10.8.11.25@o2ib6) Sep 15 06:29:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 73bd384a-ce4f-ed45-ddd4-df4c88792f0e (at 10.8.10.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe721c400, cur 1537018181 expire 1537018031 last 1537017954 Sep 15 06:35:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1055a741-3e70-fe8e-c702-1f0da06b99a7 (at 10.9.104.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b64fc00, cur 1537018544 expire 1537018394 last 1537018317 Sep 15 06:41:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 292ad22e-3a88-81e5-4018-e3d4be9f32c8 (at 10.9.102.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203ab2a800, cur 1537018865 expire 1537018715 last 1537018638 Sep 15 06:41:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 06:43:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7a29b7ea-27b6-3537-6761-540c92a0a76e (at 10.8.20.31@o2ib6) Sep 15 06:45:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f6d20838-425e-345c-498d-9342fe30287a (at 10.9.106.52@o2ib4) Sep 15 06:45:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7896719e-213b-225c-de9a-0b3a69876314 (at 10.9.101.53@o2ib4) Sep 15 06:48:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05903a8f-5c9a-bea6-dc1b-d3fe90a556ec (at 10.8.12.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fff6800, cur 1537019293 expire 1537019143 last 1537019066 Sep 15 06:48:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 06:48:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 75ec0917-15a4-7faf-a510-622c5966890d (at 10.8.10.26@o2ib6) Sep 15 06:57:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1055a741-3e70-fe8e-c702-1f0da06b99a7 (at 10.9.104.60@o2ib4) Sep 15 06:58:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9dc267dc-db8f-16fc-026a-80fb4bf407a9 (at 10.9.105.54@o2ib4) Sep 15 07:02:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ffb714a8-c8a1-cad9-be60-81f28fcc8e83 (at 10.8.20.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88057bcdb400, cur 1537020149 expire 1537019999 last 1537019922 Sep 15 07:02:29 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 07:02:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 73bd384a-ce4f-ed45-ddd4-df4c88792f0e (at 10.8.10.8@o2ib6) Sep 15 07:10:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a2ae4c69-d8f4-5e07-dd5a-faad2385cbc1 (at 10.8.13.15@o2ib6) Sep 15 07:10:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 07:11:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cc4bbfbe-be5a-eedf-b8a6-c4e8ad2f35cd (at 10.8.8.36@o2ib6) Sep 15 07:14:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f847b31-ead3-0f1f-ad35-9c262bc96b44 (at 10.8.10.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103a9a1400, cur 1537020846 expire 1537020696 last 1537020619 Sep 15 07:14:06 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 07:17:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c2f942a3-8972-7e2c-0695-9608a1ed7931 (at 10.9.105.58@o2ib4) Sep 15 07:22:19 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 07:24:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 10ef3673-afdb-3e3c-0639-a686fe26d7ea (at 10.9.113.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c7b0800, cur 1537021480 expire 1537021330 last 1537021253 Sep 15 07:24:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 07:26:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 010ca787-4268-12a3-b620-f18a75701351 (at 10.9.105.52@o2ib4) Sep 15 07:26:17 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 07:35:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb35b607-6e5e-9982-db36-09536c6deedd (at 10.8.10.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880225013000, cur 1537022112 expire 1537021962 last 1537021885 Sep 15 07:36:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 31c4ea76-8d98-76b3-0d2a-f267092359fe (at 10.9.101.72@o2ib4) Sep 15 07:36:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 07:46:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Sep 15 07:46:45 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 07:53:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e495d34-fe8b-10e7-87c5-64eda366c3f9 (at 10.8.12.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881038285000, cur 1537023212 expire 1537023062 last 1537022985 Sep 15 07:53:32 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 08:00:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eb35b607-6e5e-9982-db36-09536c6deedd (at 10.8.10.11@o2ib6) Sep 15 08:00:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 08:05:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aaed9ef4-8c69-a6fe-8c76-145b121e1c26 (at 10.9.102.49@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fec3d1000, cur 1537023912 expire 1537023762 last 1537023685 Sep 15 08:05:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 08:14:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Sep 15 08:14:40 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 08:17:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client faccb2d2-270c-56e1-efb8-dd811914efa8 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804ae1c9000, cur 1537024626 expire 1537024476 last 1537024399 Sep 15 08:17:06 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 08:27:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Sep 15 08:27:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 08:35:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef5987d3-a3e3-ac25-d801-021c3197b71d (at 10.8.2.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101b68f000, cur 1537025741 expire 1537025591 last 1537025514 Sep 15 08:35:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 08:39:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b75866aa-418d-e1f3-5cbd-48d4dc354fb1 (at 10.8.10.9@o2ib6) Sep 15 08:39:17 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 08:47:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 257efd27-5e31-fd7a-e365-12089a4f4223 (at 10.8.29.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015ca5800, cur 1537026434 expire 1537026284 last 1537026207 Sep 15 08:47:14 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 08:54:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1b79d1e8-945a-b9cf-fb85-e5f9e3598a24 (at 10.8.2.28@o2ib6) Sep 15 08:54:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 09:04:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 84cacbcf-87f9-6433-1c84-89e1b90ae6b9 (at 10.8.30.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff6153800, cur 1537027483 expire 1537027333 last 1537027256 Sep 15 09:04:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 09:10:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 09:10:12 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 15 09:22:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d2a79010-f837-c196-312f-9c244df5f9d0 (at 10.8.21.30@o2ib6) Sep 15 09:22:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 09:25:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9b520a7-2fe2-16e1-ae5c-5ab68053285a (at 10.9.101.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565cf800, cur 1537028720 expire 1537028570 last 1537028493 Sep 15 09:25:20 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Sep 15 09:26:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 637a5507-cffc-c4d6-995e-158d73c4f752 (at 10.8.17.21@o2ib6) in 152 seconds. I think it's dead, and I am evicting it. exp ffff880fd869d400, cur 1537028796 expire 1537028646 last 1537028644 Sep 15 09:32:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9372f02-f97a-291c-1e07-97d4f8f3cb2e (at 10.8.26.25@o2ib6) Sep 15 09:32:50 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Sep 15 09:33:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ff1edef3-0f3f-96c4-8ec4-14e922d3b712 (at 10.8.20.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe4440000, cur 1537029209 expire 1537029059 last 1537028982 Sep 15 09:42:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fbd4bf31-59b4-3233-305a-e8561d53fedf (at 10.8.20.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092ea02c00, cur 1537029732 expire 1537029582 last 1537029505 Sep 15 09:42:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 09:46:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 637a5507-cffc-c4d6-995e-158d73c4f752 (at 10.8.17.21@o2ib6) Sep 15 09:46:08 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 09:56:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10679be3-3dc2-6456-0b34-a512ebb5c982 (at 10.8.10.15@o2ib6) Sep 15 09:56:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 10:08:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbdf5d16-42da-afcc-a6a4-e938f9885d13 (at 10.8.13.19@o2ib6) Sep 15 10:16:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9d294355-5de1-5391-a51f-91e6f8ce1ccb (at 10.9.107.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001d9f400, cur 1537031769 expire 1537031619 last 1537031542 Sep 15 10:16:09 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 10:31:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4eb6bc82-54fa-8580-5980-110382653eaf (at 10.9.106.14@o2ib4) Sep 15 10:31:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 10:32:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3a63987-5085-63f8-b5e2-716cf51778c1 (at 10.9.104.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565c9800, cur 1537032763 expire 1537032613 last 1537032536 Sep 15 10:33:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d294355-5de1-5391-a51f-91e6f8ce1ccb (at 10.9.107.23@o2ib4) Sep 15 10:39:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 60ea0dcc-325c-4553-3aa8-fd3281ffd73a (at 10.9.107.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880180b5e400, cur 1537033167 expire 1537033017 last 1537032940 Sep 15 10:39:27 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Sep 15 10:45:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d8472f0f-2bab-481c-0a1f-2d10a2e0b3b5 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fe22c00, cur 1537033508 expire 1537033358 last 1537033281 Sep 15 10:47:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9ebd175-47c5-a73c-a10b-9bb5e732c7b8 (at 10.9.106.27@o2ib4) Sep 15 10:47:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0b90d1c8-7895-ab89-147d-433deeaac001 (at 10.9.106.22@o2ib4) Sep 15 10:48:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ece08209-1105-3335-e146-2f99aaea151d (at 10.8.18.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9447400, cur 1537033715 expire 1537033565 last 1537033488 Sep 15 10:54:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to da978049-5a18-c099-de4d-be78ebfdc922 (at 10.9.104.56@o2ib4) Sep 15 10:54:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 03be79dc-6843-d91f-4518-15178cd82d7b (at 10.9.101.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202af7e800, cur 1537034097 expire 1537033947 last 1537033870 Sep 15 10:55:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b1609ca4-997c-66b4-96d8-c6982da37213 (at 10.9.106.15@o2ib4) Sep 15 10:55:43 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Sep 15 11:01:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5fdd0ae6-89d9-64b4-5a32-3dd5f927f55c (at 10.9.105.30@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e781c00, cur 1537034492 expire 1537034342 last 1537034265 Sep 15 11:05:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 60ea0dcc-325c-4553-3aa8-fd3281ffd73a (at 10.9.107.24@o2ib4) Sep 15 11:05:42 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 11:14:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 62cb4ad8-1bd7-8935-fc78-77358d9e47db (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f3d884800, cur 1537035256 expire 1537035106 last 1537035029 Sep 15 11:15:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 03be79dc-6843-d91f-4518-15178cd82d7b (at 10.9.101.21@o2ib4) Sep 15 11:27:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aea5f6db-96e6-c50b-3565-90a7c13a756d (at 10.9.101.71@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6cc6bc00, cur 1537036066 expire 1537035916 last 1537035839 Sep 15 11:29:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 11:29:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 11:48:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aea5f6db-96e6-c50b-3565-90a7c13a756d (at 10.9.101.71@o2ib4) Sep 15 11:48:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 12:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 12:02:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 12:02:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b9b2b05-2b1a-5476-34f7-33c709a6e5bd (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018dd70c00, cur 1537038145 expire 1537037995 last 1537037918 Sep 15 12:02:25 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 12:03:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5cbf5f44-df7f-1d85-82ba-e8c5dbdf31ae (at 10.9.105.31@o2ib4) in 160 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e5b800, cur 1537038221 expire 1537038071 last 1537038061 Sep 15 12:07:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 09224f28-d4e9-41eb-4826-893251222ec0 (at 10.8.9.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe444c400, cur 1537038434 expire 1537038284 last 1537038207 Sep 15 12:14:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 163f9150-1d1f-91c2-9c07-e4bd9f0a0f81 (at 10.8.26.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880180b5d800, cur 1537038849 expire 1537038699 last 1537038622 Sep 15 12:14:09 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 12:22:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 12:25:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8288131e-d200-fbb1-8875-77d2151e3b0b (at 10.8.16.2@o2ib6) Sep 15 12:28:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8dbd6139-2097-ab63-d9a3-e6c5d4e12db0 (at 10.9.114.6@o2ib4) Sep 15 12:28:29 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 12:38:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 745fd9a9-4789-0cf2-6430-26c90ed54e2e (at 10.9.101.3@o2ib4) Sep 15 12:38:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 12:38:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c4a419f-b4c9-fb27-c17e-e54586c35016 (at 10.9.105.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85a8c000, cur 1537040335 expire 1537040185 last 1537040108 Sep 15 12:38:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 12:40:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 660a04f6-8dd6-6674-e026-482b132661fa (at 10.8.4.15@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff880e20d7c400, cur 1537040411 expire 1537040261 last 1537040240 Sep 15 12:40:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 13:00:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2c4a419f-b4c9-fb27-c17e-e54586c35016 (at 10.9.105.51@o2ib4) Sep 15 13:00:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 13:03:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 351c34c2-c0eb-6fe9-fe49-fe9e7d126c96 (at 10.9.101.1@o2ib4) Sep 15 13:17:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d72d812-c7d5-7994-46e8-37cc30b02c3b (at 10.9.113.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809717de400, cur 1537042635 expire 1537042485 last 1537042408 Sep 15 13:17:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 13:18:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 21d6fdc2-238b-fea6-0e18-6a165c1ea373 (at 10.9.101.13@o2ib4) in 194 seconds. I think it's dead, and I am evicting it. exp ffff880a76aba000, cur 1537042711 expire 1537042561 last 1537042517 Sep 15 13:20:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 13:28:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 500e20d2-ce5a-5785-91e4-158c295c1385 (at 10.9.105.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab3ef7c00, cur 1537043292 expire 1537043142 last 1537043065 Sep 15 13:28:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 13:30:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f9f8903b-0cd9-810a-179c-528ea7dfa54b (at 10.8.27.13@o2ib6) Sep 15 13:31:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b814e116-388b-51ac-1e2d-a340739837d4 (at 10.8.27.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7e211400, cur 1537043482 expire 1537043332 last 1537043255 Sep 15 13:31:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 13:32:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d72d812-c7d5-7994-46e8-37cc30b02c3b (at 10.9.113.10@o2ib4) Sep 15 13:34:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f8308a43-5f04-2f9b-181a-3d2c08b93cd1 (at 10.8.8.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102ee62800, cur 1537043686 expire 1537043536 last 1537043459 Sep 15 13:39:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 21d6fdc2-238b-fea6-0e18-6a165c1ea373 (at 10.9.101.13@o2ib4) Sep 15 13:40:04 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 13:40:05 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 13:40:05 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 15 13:42:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55573e92-b0de-7f04-62fc-6867bf399d54 (at 10.9.107.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3bb400, cur 1537044128 expire 1537043978 last 1537043901 Sep 15 13:42:08 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 13:42:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 13:49:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 500e20d2-ce5a-5785-91e4-158c295c1385 (at 10.9.105.55@o2ib4) Sep 15 13:49:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 13:53:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client abf996e5-7087-4ea9-2729-a23934548a3d (at 10.9.105.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef287bc00, cur 1537044811 expire 1537044661 last 1537044584 Sep 15 13:53:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 13:59:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8308a43-5f04-2f9b-181a-3d2c08b93cd1 (at 10.8.8.8@o2ib6) Sep 15 13:59:38 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 14:04:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 44ea970a-ec7d-41da-78ea-b4cbeff01130 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5bc59800, cur 1537045471 expire 1537045321 last 1537045244 Sep 15 14:04:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 14:05:47 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 14:14:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c48cbcc5-1a1a-de6d-a5ed-84c526bf2da1 (at 10.8.21.18@o2ib6) Sep 15 14:14:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 14:18:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a736566d-c2a6-0609-fac3-21b3e8b986e9 (at 10.9.105.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b723400, cur 1537046312 expire 1537046162 last 1537046085 Sep 15 14:36:04 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 14:37:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 363fdcba-500f-edf1-52a8-fb7a1245467b (at 10.8.22.29@o2ib6) Sep 15 14:37:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 14:38:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b54aa4b4-f2cc-6feb-c3eb-1faa5de88028 (at 10.9.101.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a37b000, cur 1537047538 expire 1537047388 last 1537047311 Sep 15 14:38:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 14:40:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a736566d-c2a6-0609-fac3-21b3e8b986e9 (at 10.9.105.53@o2ib4) Sep 15 14:47:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c7f7b469-0771-3203-e1a4-073b0d6d3f7c (at 10.9.101.63@o2ib4) Sep 15 14:48:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c7f7b469-0771-3203-e1a4-073b0d6d3f7c (at 10.9.101.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202af7d400, cur 1537048092 expire 1537047942 last 1537047865 Sep 15 14:57:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4713e175-d67a-1e95-ff61-308c94a90d2b (at 10.9.101.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe535d000, cur 1537048634 expire 1537048484 last 1537048407 Sep 15 14:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b54aa4b4-f2cc-6feb-c3eb-1faa5de88028 (at 10.9.101.16@o2ib4) Sep 15 15:04:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac246792-3f50-f88a-fe6e-c826136939c9 (at 10.9.105.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b721400, cur 1537049060 expire 1537048910 last 1537048833 Sep 15 15:10:47 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 15:14:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e09427b-1a77-bb55-d183-651631cae4a0 (at 10.8.20.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe69d8c00, cur 1537049696 expire 1537049546 last 1537049469 Sep 15 15:14:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 15:18:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4713e175-d67a-1e95-ff61-308c94a90d2b (at 10.9.101.25@o2ib4) Sep 15 15:27:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2d46e74-fc96-6b71-e3da-dfa4a99ebaf3 (at 10.9.101.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e786800, cur 1537050447 expire 1537050297 last 1537050220 Sep 15 15:29:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9633d7d-b912-38a2-907d-67f82248923b (at 10.8.26.14@o2ib6) Sep 15 15:29:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 15:42:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94e1ec08-e85e-e2aa-7a77-46e94e51b37c (at 10.8.4.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000283e800, cur 1537051366 expire 1537051216 last 1537051139 Sep 15 15:42:46 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 15:48:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e2d46e74-fc96-6b71-e3da-dfa4a99ebaf3 (at 10.9.101.52@o2ib4) Sep 15 15:48:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 15:57:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 40aa2135-b5f8-d649-a1b9-2d0fe1d59322 (at 10.9.107.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e5f800, cur 1537052226 expire 1537052076 last 1537051999 Sep 15 15:57:06 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 16:07:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f48ce40f-09bb-ec80-7163-03d5337353a5 (at 10.9.101.38@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe5359400, cur 1537052877 expire 1537052727 last 1537052650 Sep 15 16:11:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5aa908f9-3d74-45c4-538d-177880e6dd83 (at 10.8.30.32@o2ib6) Sep 15 16:11:48 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 16:13:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e500cf69-f65b-17e9-a5b0-d0f32692a333 (at 10.9.101.24@o2ib4) Sep 15 16:27:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e049ea5f-0cb4-c9ef-4754-15c742dd3322 (at 10.9.102.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101b68f800, cur 1537054067 expire 1537053917 last 1537053840 Sep 15 16:27:47 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 16:28:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f48ce40f-09bb-ec80-7163-03d5337353a5 (at 10.9.101.38@o2ib4) Sep 15 16:30:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 74957cfa-58d3-37c5-2dfa-290aa89dcd30 (at 10.9.101.14@o2ib4) Sep 15 16:35:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27064999-83c5-825b-e9a8-aafd6dead497 (at 10.9.101.15@o2ib4) Sep 15 16:47:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c74f4773-a574-536e-a978-ad56925c5fba (at 10.8.27.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be64400, cur 1537055247 expire 1537055097 last 1537055020 Sep 15 16:47:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 16:48:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e049ea5f-0cb4-c9ef-4754-15c742dd3322 (at 10.9.102.18@o2ib4) Sep 15 16:52:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 946940ec-c71f-9ee7-5c8b-23830a30b395 (at 10.9.102.38@o2ib4) Sep 15 16:52:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 16:58:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bf0fb52f-d283-67ab-2acb-580c45fa616c (at 10.9.101.4@o2ib4) Sep 15 16:59:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb00c1ca-a02c-8822-a1bb-ad0f0429fe24 (at 10.9.101.47@o2ib4) Sep 15 17:02:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 83956553-e1a8-658b-e7d0-7b4ede503451 (at 10.9.107.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a56c00, cur 1537056167 expire 1537056017 last 1537055940 Sep 15 17:02:47 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 17:06:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c74f4773-a574-536e-a978-ad56925c5fba (at 10.8.27.25@o2ib6) Sep 15 17:13:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acc0c4e8-f0db-722a-6e1c-888f2ee68ba4 (at 10.9.102.43@o2ib4) Sep 15 17:26:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 989d2e8f-13ec-a59c-a9bd-e2c728310cf1 (at 10.9.115.5@o2ib4) Sep 15 17:26:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 17:33:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b26e5ead-d1ae-e864-e4e2-4e4a1b5d2ba7 (at 10.9.106.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f5294bc00, cur 1537058034 expire 1537057884 last 1537057807 Sep 15 17:33:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 17:35:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c74371a-aefb-7af9-ead7-ed692df4ab95 (at 10.8.30.14@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff881e72c50c00, cur 1537058110 expire 1537057960 last 1537057884 Sep 15 17:41:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6bc78180-e0a2-c4aa-99fb-31d9b72dd248 (at 10.8.10.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe69d9c00, cur 1537058518 expire 1537058368 last 1537058291 Sep 15 17:53:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2c74371a-aefb-7af9-ead7-ed692df4ab95 (at 10.8.30.14@o2ib6) Sep 15 17:53:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 17:56:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f6d20838-425e-345c-498d-9342fe30287a (at 10.9.106.52@o2ib4) Sep 15 17:57:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65284ce8-3122-2e85-f967-23319d09e865 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803d1776800, cur 1537059458 expire 1537059308 last 1537059231 Sep 15 18:04:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 11ee4580-5b30-fe43-9ad2-7c756a601941 (at 10.8.10.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f80400, cur 1537059870 expire 1537059720 last 1537059643 Sep 15 18:05:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6bc78180-e0a2-c4aa-99fb-31d9b72dd248 (at 10.8.10.23@o2ib6) Sep 15 18:15:58 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 18:16:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 15 18:22:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5a8086e4-8a62-59df-b005-a83d28a6980e (at 10.9.104.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b7d9c00, cur 1537060973 expire 1537060823 last 1537060746 Sep 15 18:22:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 18:26:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to efdeba46-8c0d-ccbe-eac1-6ff8fd5378d2 (at 10.9.101.31@o2ib4) Sep 15 18:26:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 18:37:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 444f9f43-27c7-c269-1e94-f2d3b70a04d5 (at 10.8.2.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f5294f000, cur 1537061843 expire 1537061693 last 1537061616 Sep 15 18:37:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 11ee4580-5b30-fe43-9ad2-7c756a601941 (at 10.8.10.7@o2ib6) Sep 15 18:56:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74d5405b-4634-c0c0-d878-e687f85c7ee0 (at 10.9.105.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101b68e000, cur 1537062979 expire 1537062829 last 1537062752 Sep 15 19:02:24 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 15 19:02:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0fc3c314-e03c-b178-7e24-1ae4ccb8f808 (at 10.8.4.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882002f79000, cur 1537063370 expire 1537063220 last 1537063143 Sep 15 19:07:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 702cf118-d587-206a-208a-9ed3f707a5d0 (at 10.9.101.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880963363800, cur 1537063633 expire 1537063483 last 1537063406 Sep 15 19:07:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 19:08:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea9b8738-e6bf-c343-cf36-b2d7da014b4c (at 10.8.10.24@o2ib6) in 203 seconds. I think it's dead, and I am evicting it. exp ffff880caff5c400, cur 1537063709 expire 1537063559 last 1537063506 Sep 15 19:09:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8b826015-ca49-b9c4-2bad-3ad8777ef035 (at 10.9.102.68@o2ib4) in 181 seconds. I think it's dead, and I am evicting it. exp ffff88203a2f2400, cur 1537063785 expire 1537063635 last 1537063604 Sep 15 19:13:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0de37dcd-f8ba-0a3c-681f-e8335deb896a (at 10.8.4.21@o2ib6) Sep 15 19:13:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 19:17:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 74d5405b-4634-c0c0-d878-e687f85c7ee0 (at 10.9.105.3@o2ib4) Sep 15 19:18:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f5a1a98a-f3c9-c19e-83f9-874ab3c48e89 (at 10.8.12.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9444800, cur 1537064336 expire 1537064186 last 1537064109 Sep 15 19:27:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 291d7bea-f250-4d15-130f-a2b94cc43882 (at 10.9.101.30@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202af7d800, cur 1537064837 expire 1537064687 last 1537064610 Sep 15 19:28:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 702cf118-d587-206a-208a-9ed3f707a5d0 (at 10.9.101.9@o2ib4) Sep 15 19:28:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 19:36:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0fc3c314-e03c-b178-7e24-1ae4ccb8f808 (at 10.8.4.1@o2ib6) Sep 15 19:36:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 19:37:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e6855f59-3e88-618d-b247-85953673194b (at 10.9.105.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b721c00, cur 1537065477 expire 1537065327 last 1537065250 Sep 15 19:48:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 291d7bea-f250-4d15-130f-a2b94cc43882 (at 10.9.101.30@o2ib4) Sep 15 19:48:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 19:58:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47dcea70-e6ea-f709-7a75-8991089fd9b8 (at 10.8.2.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f0d723000, cur 1537066686 expire 1537066536 last 1537066459 Sep 15 20:00:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e6855f59-3e88-618d-b247-85953673194b (at 10.9.105.20@o2ib4) Sep 15 20:00:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f066dded-f6ec-4818-731a-b0e1cfb98fd3 (at 10.8.22.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd869fc00, cur 1537066857 expire 1537066707 last 1537066630 Sep 15 20:00:57 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 15 20:04:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93942065-09d1-8d73-551f-ad78518ec0e9 (at 10.9.102.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801393cdc00, cur 1537067067 expire 1537066917 last 1537066840 Sep 15 20:17:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d049b869-d728-0a9c-64a8-4c2cb1a8428f (at 10.8.3.7@o2ib6) Sep 15 20:17:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 20:19:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 23fcfacf-d0bc-678e-64c4-8bad17b50271 (at 10.9.101.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6cc6f800, cur 1537067998 expire 1537067848 last 1537067771 Sep 15 20:27:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 50b5e033-e287-5c20-b2f0-b8f9cf4514a4 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d6e211800, cur 1537068472 expire 1537068322 last 1537068245 Sep 15 20:27:52 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 15 20:35:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9573b08a-53b0-fb7f-8bbe-fd5b344c0bad (at 10.9.105.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e7be400, cur 1537068948 expire 1537068798 last 1537068721 Sep 15 20:43:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 33818e61-1c0a-031d-ad44-7a27633749c0 (at 10.8.3.29@o2ib6) Sep 15 20:43:49 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Sep 15 20:44:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f3b2655-2871-d390-dbf3-fb6d02454dc8 (at 10.8.13.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a106a7800, cur 1537069454 expire 1537069304 last 1537069227 Sep 15 20:57:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9573b08a-53b0-fb7f-8bbe-fd5b344c0bad (at 10.9.105.59@o2ib4) Sep 15 20:57:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 15 20:59:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c328a9ac-da17-ec8e-0947-0f682a2ee0ef (at 10.8.26.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe5bdcc00, cur 1537070373 expire 1537070223 last 1537070146 Sep 15 20:59:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 21:02:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f3b2655-2871-d390-dbf3-fb6d02454dc8 (at 10.8.13.28@o2ib6) Sep 15 21:03:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7f488842-06e4-96d4-ad37-762861415824 (at 10.8.21.10@o2ib6) Sep 15 21:09:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2f17349-9e3d-3017-bc77-7f26ebade7b8 (at 10.8.11.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f63bc00, cur 1537070947 expire 1537070797 last 1537070720 Sep 15 21:13:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be9cd585-2919-8988-d157-352d89cf6682 (at 10.9.102.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200ea66800, cur 1537071221 expire 1537071071 last 1537070994 Sep 15 21:17:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c328a9ac-da17-ec8e-0947-0f682a2ee0ef (at 10.8.26.10@o2ib6) Sep 15 21:23:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59ac0e13-2392-d6f9-ff08-5ea7e80526c7 (at 10.8.14.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f62574000, cur 1537071838 expire 1537071688 last 1537071611 Sep 15 21:24:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6f11ebc-0bd6-ad8b-390a-e041d9836de8 (at 10.9.115.6@o2ib4) Sep 15 21:25:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7f7e432-db3d-fd29-36c0-946bc794ceed (at 10.9.102.31@o2ib4) in 217 seconds. I think it's dead, and I am evicting it. exp ffff881fec3d3000, cur 1537071914 expire 1537071764 last 1537071697 Sep 15 21:25:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 21:32:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e2f17349-9e3d-3017-bc77-7f26ebade7b8 (at 10.8.11.6@o2ib6) Sep 15 21:35:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to be9cd585-2919-8988-d157-352d89cf6682 (at 10.9.102.23@o2ib4) Sep 15 21:39:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2104b814-a56f-f122-8c9f-eb491e58982b (at 10.8.12.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092ea07c00, cur 1537072774 expire 1537072624 last 1537072547 Sep 15 21:48:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e7f7e432-db3d-fd29-36c0-946bc794ceed (at 10.9.102.31@o2ib4) Sep 15 21:52:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e4a427d8-76fe-0d6e-ff77-7dbb5f14bea9 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aedf7d800, cur 1537073557 expire 1537073407 last 1537073330 Sep 15 21:52:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 21:53:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 22:03:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2104b814-a56f-f122-8c9f-eb491e58982b (at 10.8.12.33@o2ib6) Sep 15 22:07:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 04a16594-4e3a-e006-2ee5-05d680e0771c (at 10.8.23.13@o2ib6) Sep 15 22:09:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e8f53bba-4951-83a9-2815-fe1bb258ac67 (at 10.8.11.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102a2f8800, cur 1537074546 expire 1537074396 last 1537074319 Sep 15 22:24:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f5dd0e62-18e3-7763-c12e-e4abb0f5f1e6 (at 10.8.4.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096d3f0000, cur 1537075467 expire 1537075317 last 1537075240 Sep 15 22:28:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 22:29:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51b8199a-2088-48d4-e7e1-f8961b9f4bd3 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1f800, cur 1537075746 expire 1537075596 last 1537075519 Sep 15 22:29:06 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 15 22:33:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e8f53bba-4951-83a9-2815-fe1bb258ac67 (at 10.8.11.31@o2ib6) Sep 15 22:35:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f4f01e7-e055-e4fe-1c7f-c730a83fa6c5 (at 10.8.13.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff6154000, cur 1537076145 expire 1537075995 last 1537075918 Sep 15 22:45:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 583f3ef7-dfbf-e305-3ba8-2db1fc44b331 (at 10.9.101.44@o2ib4) Sep 15 22:45:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14b712f1-d538-732a-e3dd-2e88d765d536 (at 10.9.101.42@o2ib4) Sep 15 22:45:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6fe5ed74-50cb-72b3-2447-0b5a4d58c709 (at 10.9.101.56@o2ib4) Sep 15 22:49:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f5dd0e62-18e3-7763-c12e-e4abb0f5f1e6 (at 10.8.4.30@o2ib6) Sep 15 22:49:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 22:54:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8f4f01e7-e055-e4fe-1c7f-c730a83fa6c5 (at 10.8.13.25@o2ib6) Sep 15 22:57:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ce570b4-4160-fcb0-01fd-67c1f4bdd8e2 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804ae1cd400, cur 1537077437 expire 1537077287 last 1537077210 Sep 15 22:57:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 22:57:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 15 23:05:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f6f4b4e6-b35e-c489-19e3-34ffdfd88154 (at 10.8.30.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd7a78000, cur 1537077929 expire 1537077779 last 1537077702 Sep 15 23:06:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43241e91-cff0-437e-bacc-48942c957e6a (at 10.9.101.51@o2ib4) Sep 15 23:06:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43241e91-cff0-437e-bacc-48942c957e6a (at 10.9.101.51@o2ib4) in 207 seconds. I think it's dead, and I am evicting it. exp ffff880180b5e800, cur 1537078005 expire 1537077855 last 1537077798 Sep 15 23:06:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 23:13:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f6f4b4e6-b35e-c489-19e3-34ffdfd88154 (at 10.8.30.1@o2ib6) Sep 15 23:15:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f3ec79b2-4842-1d7f-7e69-8775f4253bb2 (at 10.9.101.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85a8a800, cur 1537078520 expire 1537078370 last 1537078293 Sep 15 23:27:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65d53997-49a8-434d-70f4-bf7febde2942 (at 10.9.106.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800280d6000, cur 1537079243 expire 1537079093 last 1537079016 Sep 15 23:27:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 23:35:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f3ec79b2-4842-1d7f-7e69-8775f4253bb2 (at 10.9.101.45@o2ib4) Sep 15 23:35:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 23:40:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 96afcbea-9310-e67f-1fe7-5fae0374f13b (at 10.9.106.25@o2ib4) Sep 15 23:40:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 15 23:48:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d6bf9621-b36e-0c21-a19b-641c9a6f51cf (at 10.9.104.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca48000, cur 1537080485 expire 1537080335 last 1537080258 Sep 15 23:50:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0fc1e0c6-9efd-dc0b-4fdd-6e8767210cfa (at 10.8.1.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc169fc00, cur 1537080650 expire 1537080500 last 1537080423 Sep 15 23:59:16 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 00:00:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2e0a1998-f8f4-b6ec-c9f1-ed0a3b20546e (at 10.8.1.20@o2ib6) Sep 16 00:04:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 415c02a0-7be6-dd07-c9f5-5b30f452d4ee (at 10.9.105.36@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b64ac00, cur 1537081451 expire 1537081301 last 1537081224 Sep 16 00:09:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3184b540-a723-dd81-7aca-d3bcdd522749 (at 10.8.10.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ed3538800, cur 1537081760 expire 1537081610 last 1537081533 Sep 16 00:09:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 00:09:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d6bf9621-b36e-0c21-a19b-641c9a6f51cf (at 10.9.104.18@o2ib4) Sep 16 00:13:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7311cb7-4a52-7fe4-7e17-432ae5dfbf4b (at 10.8.27.22@o2ib6) Sep 16 00:25:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 415c02a0-7be6-dd07-c9f5-5b30f452d4ee (at 10.9.105.36@o2ib4) Sep 16 00:33:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3184b540-a723-dd81-7aca-d3bcdd522749 (at 10.8.10.3@o2ib6) Sep 16 00:39:17 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 01:18:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4434a34a-1546-ee9c-7b52-8eef40362eab (at 10.8.3.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881030656000, cur 1537085910 expire 1537085760 last 1537085683 Sep 16 01:19:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9d459274-c074-0147-307d-ea1288dc6a7c (at 10.9.107.17@o2ib4) in 171 seconds. I think it's dead, and I am evicting it. exp ffff882001d9c800, cur 1537085986 expire 1537085836 last 1537085815 Sep 16 01:19:46 oak-md1-s2 kernel: Lustre: Skipped 19 previous similar messages Sep 16 01:25:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3937698c-9eda-0b14-554b-891121063a0e (at 10.12.4.26@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be02800, cur 1537086300 expire 1537086150 last 1537086073 Sep 16 01:27:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3937698c-9eda-0b14-554b-891121063a0e (at 10.12.4.26@o2ib) Sep 16 01:28:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b4fe4e8a-a292-d39b-efe4-3c357f0f782d (at 10.8.3.36@o2ib6) Sep 16 01:28:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59fa84eb-a290-413e-110d-b73eaa7c3ef7 (at 10.8.6.6@o2ib6) Sep 16 01:28:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 01:30:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0d3c3008-f1e2-87cc-f26c-8febfe57913d (at 10.8.6.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cde68f000, cur 1537086615 expire 1537086465 last 1537086388 Sep 16 01:33:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4e86e85-f54f-00e9-0d8f-bb1ecc03b43e (at 10.8.3.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2cc800, cur 1537086823 expire 1537086673 last 1537086596 Sep 16 01:33:43 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 16 01:34:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c0d54f67-0651-3609-cd1b-a13bb47d3d78 (at 10.8.3.8@o2ib6) Sep 16 01:34:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 01:39:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fd226631-ec42-ea48-1eb1-b60097c20b51 (at 10.9.101.55@o2ib4) Sep 16 01:40:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ee4f455-f50b-86d3-08ec-6df062d9d382 (at 10.8.2.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab4309400, cur 1537087223 expire 1537087073 last 1537086996 Sep 16 01:40:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 01:41:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eaee074c-c42a-1b0d-d24f-6a4e0ab844fd (at 10.8.3.10@o2ib6) Sep 16 01:42:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 611efb8d-b13f-514f-89b1-19c11248b03b (at 10.8.3.11@o2ib6) Sep 16 01:43:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cde12fa6-db73-c505-158a-513511bfbb56 (at 10.9.104.68@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef287dc00, cur 1537087416 expire 1537087266 last 1537087189 Sep 16 01:43:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4434a34a-1546-ee9c-7b52-8eef40362eab (at 10.8.3.25@o2ib6) Sep 16 01:43:37 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 16 01:44:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) in 151 seconds. I think it's dead, and I am evicting it. exp ffff881fe4446800, cur 1537087492 expire 1537087342 last 1537087341 Sep 16 01:46:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d459274-c074-0147-307d-ea1288dc6a7c (at 10.9.107.17@o2ib4) Sep 16 01:46:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 01:46:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 077a3301-01ad-25c0-566b-6db505f05b01 (at 10.8.3.35@o2ib6) in 174 seconds. I think it's dead, and I am evicting it. exp ffff880b61433c00, cur 1537087568 expire 1537087418 last 1537087394 Sep 16 01:50:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4ee4f455-f50b-86d3-08ec-6df062d9d382 (at 10.8.2.23@o2ib6) Sep 16 01:53:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 32b9b8b9-b19c-790e-c493-0374cb677e13 (at 10.9.115.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c588000, cur 1537088034 expire 1537087884 last 1537087807 Sep 16 01:53:54 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 16 01:58:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 308144a3-6174-b4e7-134e-05b0cb8fda0d (at 10.8.3.18@o2ib6) Sep 16 01:58:29 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 16 02:11:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 077a3301-01ad-25c0-566b-6db505f05b01 (at 10.8.3.35@o2ib6) Sep 16 02:11:11 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 16 02:18:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f8533e8b-3a11-3aef-6b11-f9c53bf9ac57 (at 10.8.27.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019e616800, cur 1537089520 expire 1537089370 last 1537089293 Sep 16 02:21:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ef27a59f-0454-6a00-33a2-aaddfb64befe (at 10.8.4.6@o2ib6) Sep 16 02:21:20 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 16 02:37:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8533e8b-3a11-3aef-6b11-f9c53bf9ac57 (at 10.8.27.21@o2ib6) Sep 16 02:47:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 92e3c2c5-544d-c491-cc21-b2824f61dca9 (at 10.8.11.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f710000, cur 1537091264 expire 1537091114 last 1537091037 Sep 16 02:49:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8510168d-083c-4068-bd1f-5511c2aefd29 (at 10.8.11.12@o2ib6) in 175 seconds. I think it's dead, and I am evicting it. exp ffff881ea8c46c00, cur 1537091340 expire 1537091190 last 1537091165 Sep 16 02:57:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e17d9d800, cur 1537091864 expire 1537091714 last 1537091637 Sep 16 02:59:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5c544306-e891-15cd-9066-333a1007d34d (at 10.8.11.4@o2ib6) Sep 16 03:28:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Sep 16 03:28:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 03:40:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 432ad4b2-7729-0ed2-084d-f39675532388 (at 10.8.6.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da7a76000, cur 1537094451 expire 1537094301 last 1537094224 Sep 16 04:05:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0f62ae40-e58e-d7bc-6c96-d5eec9ff1569 (at 10.8.6.22@o2ib6) Sep 16 04:20:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b22da56b-799d-9466-de3c-65a7fc76165d (at 10.8.17.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11863800, cur 1537096809 expire 1537096659 last 1537096582 Sep 16 04:28:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 819c5dc0-37df-8204-e03c-d26068a6d625 (at 10.8.17.25@o2ib6) Sep 16 05:37:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ba7fc192-81c6-90e7-7750-28bb61bdeb07 (at 10.8.4.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001cda6400, cur 1537101473 expire 1537101323 last 1537101246 Sep 16 05:48:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ddfd0ee-8c85-e8c0-a934-802dffd88125 (at 10.8.4.17@o2ib6) Sep 16 06:00:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6cdfa4a1-0eb5-fc32-84dc-5732f06d5d85 (at 10.8.4.7@o2ib6) Sep 16 06:01:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b067ccdd-27cb-ae5c-27d9-df62cb606375 (at 10.8.4.19@o2ib6) Sep 16 06:01:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09299d5d-d5ed-d979-4e97-6a3fa86ae647 (at 10.8.4.8@o2ib6) Sep 16 06:02:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3796746c-13cb-1dff-031b-f34216a2f802 (at 10.8.4.18@o2ib6) Sep 16 06:02:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dafb6a59-e11b-03ee-af38-b4b95700f3c7 (at 10.8.4.22@o2ib6) Sep 16 06:02:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6a90cab3-cdb6-8407-e4e8-13b233049734 (at 10.8.4.20@o2ib6) Sep 16 06:06:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6289a368-dd6e-c696-8bba-52642ee3ef0d (at 10.8.1.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103890c800, cur 1537103174 expire 1537103024 last 1537102947 Sep 16 06:06:14 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 16 06:10:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9411437c-225a-bf4a-e991-7fd9cb0edb7a (at 10.8.6.31@o2ib6) Sep 16 06:12:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 91c0a371-2eb7-6af5-f927-a00df13ee31f (at 10.8.2.6@o2ib6) Sep 16 06:12:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 06:15:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6287b08e-7a2b-d21f-9862-c53903ea650f (at 10.8.1.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab3ef7000, cur 1537103736 expire 1537103586 last 1537103509 Sep 16 06:16:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6289a368-dd6e-c696-8bba-52642ee3ef0d (at 10.8.1.32@o2ib6) Sep 16 06:24:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5deefe8-6878-a928-038f-63911141e46c (at 10.8.4.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096d3f6800, cur 1537104251 expire 1537104101 last 1537104024 Sep 16 06:29:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb9e603f-bb8f-11d1-6162-4080c55179f0 (at 10.8.3.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b61433400, cur 1537104552 expire 1537104402 last 1537104325 Sep 16 06:29:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 06:39:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6287b08e-7a2b-d21f-9862-c53903ea650f (at 10.8.1.33@o2ib6) Sep 16 06:47:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c1c5ea59-40db-8ad9-fb5f-2c9fd5cf9542 (at 10.12.4.59@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be05000, cur 1537105676 expire 1537105526 last 1537105449 Sep 16 06:48:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c5deefe8-6878-a928-038f-63911141e46c (at 10.8.4.32@o2ib6) Sep 16 06:53:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb9e603f-bb8f-11d1-6162-4080c55179f0 (at 10.8.3.22@o2ib6) Sep 16 06:58:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4b119af-2df3-64a1-a0dc-9e0c140c1823 (at 10.8.4.12@o2ib6) Sep 16 07:06:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2542864-5025-7878-8d3e-769f383ab97e (at 10.8.4.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880814640800, cur 1537106801 expire 1537106651 last 1537106574 Sep 16 07:25:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ccf7dc8-0581-5140-3b74-6b0819360526 (at 10.9.105.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b64b400, cur 1537107920 expire 1537107770 last 1537107693 Sep 16 07:30:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f2542864-5025-7878-8d3e-769f383ab97e (at 10.8.4.36@o2ib6) Sep 16 07:47:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4ccf7dc8-0581-5140-3b74-6b0819360526 (at 10.9.105.26@o2ib4) Sep 16 07:58:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client feed091a-a7fc-23bb-a646-4bba0a30ce7e (at 10.9.107.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3bb800, cur 1537109905 expire 1537109755 last 1537109678 Sep 16 08:24:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to feed091a-a7fc-23bb-a646-4bba0a30ce7e (at 10.9.107.18@o2ib4) Sep 16 08:57:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102dfeb800, cur 1537113447 expire 1537113297 last 1537113220 Sep 16 09:01:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d39e137-e7fc-0b0e-ffa4-c1aa9e95dcc7 (at 10.8.11.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f86400, cur 1537113698 expire 1537113548 last 1537113471 Sep 16 09:02:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33faf189-ce4d-0062-6058-69aa36ab83be (at 10.12.4.26@o2ib) in 226 seconds. I think it's dead, and I am evicting it. exp ffff880a8657c800, cur 1537113774 expire 1537113624 last 1537113548 Sep 16 09:05:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3937698c-9eda-0b14-554b-891121063a0e (at 10.12.4.26@o2ib) Sep 16 09:15:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a9b9ea1-3786-1d90-2c15-70a68834d78c (at 10.8.3.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804bc219000, cur 1537114535 expire 1537114385 last 1537114308 Sep 16 09:18:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c1c5ea59-40db-8ad9-fb5f-2c9fd5cf9542 (at 10.12.4.59@o2ib) Sep 16 09:24:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 14b8d305-ce8c-ddcf-a413-3e35caffc922 (at 10.8.27.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063423800, cur 1537115056 expire 1537114906 last 1537114829 Sep 16 09:26:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d39e137-e7fc-0b0e-ffa4-c1aa9e95dcc7 (at 10.8.11.33@o2ib6) Sep 16 09:31:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Sep 16 09:42:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14b8d305-ce8c-ddcf-a413-3e35caffc922 (at 10.8.27.10@o2ib6) Sep 16 09:44:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9e0951a-b1a4-67a8-3107-4f9b1605d52b (at 10.8.21.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102a2fc800, cur 1537116251 expire 1537116101 last 1537116024 Sep 16 09:45:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce34dc22-8781-85f7-38cd-d61360d35a1d (at 10.9.102.9@o2ib4) in 223 seconds. I think it's dead, and I am evicting it. exp ffff881fec3d2000, cur 1537116327 expire 1537116177 last 1537116104 Sep 16 09:45:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 09:58:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 624e7777-399e-18d8-9e9d-efcfdc6c943a (at 10.8.11.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe534f800, cur 1537117102 expire 1537116952 last 1537116875 Sep 16 09:59:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b610587-01b4-c16d-f180-8e6dccea1ccc (at 10.9.101.68@o2ib4) in 191 seconds. I think it's dead, and I am evicting it. exp ffff880f6cc6f000, cur 1537117178 expire 1537117028 last 1537116987 Sep 16 10:06:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 222a85d2-e58d-79ae-2077-4b2d38a62550 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff438d800, cur 1537117573 expire 1537117423 last 1537117346 Sep 16 10:08:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ce34dc22-8781-85f7-38cd-d61360d35a1d (at 10.9.102.9@o2ib4) Sep 16 10:09:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7a799079-2cdd-bbe9-600e-180f51f22ca1 (at 10.8.1.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab430b800, cur 1537117757 expire 1537117607 last 1537117530 Sep 16 10:11:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7756668b-cf20-8015-69f5-3791fb50b137 (at 10.8.22.17@o2ib6) Sep 16 10:12:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9e0951a-b1a4-67a8-3107-4f9b1605d52b (at 10.8.21.33@o2ib6) Sep 16 10:18:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e7efa09-d456-0658-2c64-26540ce9ab0a (at 10.8.2.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809717de000, cur 1537118307 expire 1537118157 last 1537118080 Sep 16 10:20:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6b610587-01b4-c16d-f180-8e6dccea1ccc (at 10.9.101.68@o2ib4) Sep 16 10:21:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 624e7777-399e-18d8-9e9d-efcfdc6c943a (at 10.8.11.1@o2ib6) Sep 16 10:22:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Sep 16 10:33:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7a799079-2cdd-bbe9-600e-180f51f22ca1 (at 10.8.1.31@o2ib6) Sep 16 10:41:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3e7efa09-d456-0658-2c64-26540ce9ab0a (at 10.8.2.8@o2ib6) Sep 16 11:26:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 46f51a37-bd31-ab3e-e73a-5aa75fa2a817 (at 10.8.12.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102dfe8800, cur 1537122402 expire 1537122252 last 1537122175 Sep 16 11:37:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46f51a37-bd31-ab3e-e73a-5aa75fa2a817 (at 10.8.12.34@o2ib6) Sep 16 11:40:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb20c4a3-e0c7-c13a-4ce5-ce5af32e5e2d (at 10.8.7.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800500c6400, cur 1537123201 expire 1537123051 last 1537122974 Sep 16 11:46:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b8165cde-d5fa-2a3b-7949-1d2dca4dc294 (at 10.8.18.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b63e33400, cur 1537123598 expire 1537123448 last 1537123371 Sep 16 11:47:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4730725b-dee7-2762-9268-4ecf9882c03e (at 10.9.106.50@o2ib4) Sep 16 11:47:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6bb311c6-e609-ef1e-5e6a-bcc6c26e17a8 (at 10.9.106.50@o2ib4) in 200 seconds. I think it's dead, and I am evicting it. exp ffff88184a665000, cur 1537123674 expire 1537123524 last 1537123474 Sep 16 11:50:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bcd8d808-9b19-fd1e-128b-e8937d8aae22 (at 10.8.6.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a106a3c00, cur 1537123828 expire 1537123678 last 1537123601 Sep 16 11:55:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6ab0e4d4-0460-2126-aa53-54027560a860 (at 10.8.2.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005c336800, cur 1537124142 expire 1537123992 last 1537123915 Sep 16 11:55:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 11:56:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dcf75eba-1006-ff57-1378-02d2632c499a (at 10.8.17.9@o2ib6) in 197 seconds. I think it's dead, and I am evicting it. exp ffff881fe5a5a000, cur 1537124218 expire 1537124068 last 1537124021 Sep 16 12:03:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2e2a365f-3b8d-6fd1-d503-3c4e36e6c5fc (at 10.8.7.18@o2ib6) Sep 16 12:04:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 45fe9d74-6afc-e652-7268-4e9af6db4ea1 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881861760400, cur 1537124640 expire 1537124490 last 1537124413 Sep 16 12:04:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 12:05:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b946132a-b606-fbeb-48e4-bdb733b0b9f3 (at 10.8.2.32@o2ib6) Sep 16 12:05:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b8165cde-d5fa-2a3b-7949-1d2dca4dc294 (at 10.8.18.3@o2ib6) Sep 16 12:13:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d6a74f9-93bc-dd56-f9a2-3fa4a1d748e3 (at 10.9.104.10@o2ib4) Sep 16 12:14:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3b0a3a8-4dfa-9e30-032a-466e54ce313a (at 10.8.7.17@o2ib6) Sep 16 12:14:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3d6a74f9-93bc-dd56-f9a2-3fa4a1d748e3 (at 10.9.104.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe535cc00, cur 1537125291 expire 1537125141 last 1537125064 Sep 16 12:16:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 87d4e3a3-749e-3f44-048c-7d9652461f82 (at 10.8.18.33@o2ib6) Sep 16 12:18:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c3aa5cb-231e-dabf-9eb6-3dbe970f3654 (at 10.9.106.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200ea66000, cur 1537125516 expire 1537125366 last 1537125289 Sep 16 12:19:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8271ebc4-d739-add6-952f-6177412af161 (at 10.9.112.16@o2ib4) Sep 16 12:19:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 12:23:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dcf75eba-1006-ff57-1378-02d2632c499a (at 10.8.17.9@o2ib6) Sep 16 12:27:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bbc9505d-0de1-08f6-d72b-c028b41c860e (at 10.9.113.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef106000, cur 1537126059 expire 1537125909 last 1537125832 Sep 16 12:27:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a46c467a-9f53-b135-92c6-44c55506a9ad (at 10.9.113.1@o2ib4) Sep 16 12:32:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0c3aa5cb-231e-dabf-9eb6-3dbe970f3654 (at 10.9.106.8@o2ib4) Sep 16 12:40:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ba5fa55f-c7a8-5b26-b603-7c8104184808 (at 10.8.2.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea84de000, cur 1537126811 expire 1537126661 last 1537126584 Sep 16 12:56:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da29c1d7-f4e1-c2dd-0245-d7f9bebda4f5 (at 10.8.25.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809717dcc00, cur 1537127799 expire 1537127649 last 1537127572 Sep 16 13:03:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1fc5f828-1677-851c-3d57-213420629f5f (at 10.9.113.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da9112c00, cur 1537128214 expire 1537128064 last 1537127987 Sep 16 13:09:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3293e693-2489-9614-6bc0-6196f4ab5295 (at 10.8.8.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba4000, cur 1537128557 expire 1537128407 last 1537128330 Sep 16 13:09:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 36d53d1e-e3eb-2a5a-86b1-275956e5698b (at 10.8.8.15@o2ib6) Sep 16 13:12:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5be35293-3690-ba71-55ac-1eda538771b5 (at 10.8.2.3@o2ib6) Sep 16 13:14:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to da29c1d7-f4e1-c2dd-0245-d7f9bebda4f5 (at 10.8.25.24@o2ib6) Sep 16 13:15:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 703aec57-4774-10ed-b4be-e12a0fd6f9a1 (at 10.9.107.34@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881999302000, cur 1537128945 expire 1537128795 last 1537128718 Sep 16 13:19:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 208dc229-120b-fb25-969a-4251ce07291e (at 10.9.113.2@o2ib4) Sep 16 13:23:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 18cc88d0-5e73-d612-929f-2dabaf0e3e6f (at 10.8.27.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc1540400, cur 1537129428 expire 1537129278 last 1537129201 Sep 16 13:28:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f3bbafc-b541-0179-1a49-b8dda21af7e1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8c9800, cur 1537129702 expire 1537129552 last 1537129475 Sep 16 13:28:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 16 13:32:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 32e4a6f0-11fd-4c60-2a84-6b4c7133ec3a (at 10.8.17.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a28ef8400, cur 1537129947 expire 1537129797 last 1537129720 Sep 16 13:33:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2b18e4a-22f1-07e6-d38e-cc4b315586b9 (at 10.8.18.35@o2ib6) in 196 seconds. I think it's dead, and I am evicting it. exp ffff880f1708fc00, cur 1537130023 expire 1537129873 last 1537129827 Sep 16 13:41:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 83e1b268-93b0-cdeb-4bb7-f64fec1e15c7 (at 10.8.27.16@o2ib6) Sep 16 13:41:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 13:48:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c941a37e-0dd2-34bd-562b-0b88621639ad (at 10.8.7.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab2f44c00, cur 1537130906 expire 1537130756 last 1537130679 Sep 16 13:57:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 67f34637-d4a2-c6cf-82a0-85908ab6b8ab (at 10.8.7.2@o2ib6) Sep 16 13:57:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 14:07:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 77956b2b-c4cc-c7e3-e0a1-9cbad164c31b (at 10.8.8.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd941000, cur 1537132050 expire 1537131900 last 1537131823 Sep 16 14:08:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1df346e0-a291-4cc4-e722-4ee762ad59a0 (at 10.8.1.22@o2ib6) in 211 seconds. I think it's dead, and I am evicting it. exp ffff880d0947d800, cur 1537132126 expire 1537131976 last 1537131915 Sep 16 14:15:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 603d84a9-a3fe-600f-6edf-f0a946945b8c (at 10.8.8.30@o2ib6) Sep 16 14:18:09 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 14:18:10 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 14:18:10 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 16 14:18:11 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 14:18:11 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Sep 16 14:18:13 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 14:18:13 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 32 previous similar messages Sep 16 14:18:17 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 14:18:17 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 79 previous similar messages Sep 16 14:18:25 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 14:18:25 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 130 previous similar messages Sep 16 14:18:41 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 14:18:41 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 255 previous similar messages Sep 16 14:22:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3449ec0-e6fd-05c9-1da1-faeeaf95c643 (at 10.8.12.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806d99f1c00, cur 1537132978 expire 1537132828 last 1537132751 Sep 16 14:32:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f4a61592-7d40-9916-52cf-6afa19fcae00 (at 10.8.1.22@o2ib6) Sep 16 15:06:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90cca6d7-cd4e-d957-755b-ad5b95fdf2bd (at 10.8.3.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c8a7d5400, cur 1537135603 expire 1537135453 last 1537135376 Sep 16 15:11:34 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 15:11:34 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Sep 16 15:19:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3741439d-b69f-7116-f3b8-6b5b7d6861e2 (at 10.8.3.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880109888000, cur 1537136348 expire 1537136198 last 1537136121 Sep 16 15:19:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 15:29:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 16 15:29:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 15:30:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0bb2531-586c-138e-849a-c911f159bd14 (at 10.8.4.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802a5556800, cur 1537137026 expire 1537136876 last 1537136799 Sep 16 15:30:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 15:34:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34263530-dddb-d9e6-8e5d-2c189bd873b2 (at 10.8.27.33@o2ib6) Sep 16 15:34:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 15:35:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f95fd25-9e9f-0955-97fa-7533582cffc5 (at 10.8.27.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edf4c00, cur 1537137352 expire 1537137202 last 1537137125 Sep 16 15:43:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34974ccb-c1b1-bcb8-835a-466ad18e87c2 (at 10.8.3.20@o2ib6) Sep 16 15:47:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94b7244e-27b5-d598-0186-d1954ff0c52d (at 10.8.18.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea7400, cur 1537138057 expire 1537137907 last 1537137830 Sep 16 15:48:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c86ea259-be11-67ed-188f-9702bbb62f4e (at 10.8.3.21@o2ib6) in 183 seconds. I think it's dead, and I am evicting it. exp ffff88103aa1d000, cur 1537138133 expire 1537137983 last 1537137950 Sep 16 15:51:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d838111a-6033-890d-9526-a4aed4d6e6b4 (at 10.8.6.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d094bf000, cur 1537138286 expire 1537138136 last 1537138059 Sep 16 15:51:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 15:52:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d92ab997-b91d-a38d-68d8-f6b80d668b5f (at 10.8.3.23@o2ib6) Sep 16 15:52:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 15:54:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33762c6a-5979-58e4-8796-308284779b96 (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a3bec5000, cur 1537138497 expire 1537138347 last 1537138270 Sep 16 15:54:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 15:57:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8624f01e-1d3e-4405-7476-6476c124e6e7 (at 10.8.4.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a106a4800, cur 1537138652 expire 1537138502 last 1537138425 Sep 16 16:04:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 16 16:04:01 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 16 16:15:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c86ea259-be11-67ed-188f-9702bbb62f4e (at 10.8.3.21@o2ib6) Sep 16 16:15:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 16:17:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 601f9bcd-0f77-f3b2-6d5c-6492e859d2b1 (at 10.8.18.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88027da59800, cur 1537139836 expire 1537139686 last 1537139609 Sep 16 16:20:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 15ec7c10-56ee-ec2a-0465-ab8d719c7044 (at 10.9.105.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc3efd800, cur 1537140009 expire 1537139859 last 1537139782 Sep 16 16:25:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 84094d65-d965-33fa-e20e-a153e9fab037 (at 10.8.18.9@o2ib6) Sep 16 16:25:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 16:31:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bba087b5-4073-e575-2ccb-5db0f3ecda36 (at 10.8.6.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880074ead000, cur 1537140676 expire 1537140526 last 1537140449 Sep 16 16:31:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 16:35:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ae94460-7d34-fd1f-7a09-5ca0ebbb2447 (at 10.8.21.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063426000, cur 1537140922 expire 1537140772 last 1537140695 Sep 16 16:41:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 15ec7c10-56ee-ec2a-0465-ab8d719c7044 (at 10.9.105.25@o2ib4) Sep 16 16:50:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2874060-6bfb-c6c5-c8a8-60d8ca27c3be (at 10.8.6.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101bf10400, cur 1537141854 expire 1537141704 last 1537141627 Sep 16 16:53:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 69bc8ae0-d560-1e7e-c335-63319ac6c4e2 (at 10.9.101.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f80800, cur 1537142013 expire 1537141863 last 1537141786 Sep 16 16:58:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 86e86b42-a5a8-54ee-54ba-26ee9c42caf2 (at 10.8.6.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d5c9800, cur 1537142285 expire 1537142135 last 1537142058 Sep 16 16:58:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 455e8050-19a4-2e12-d889-3d939ef346af (at 10.8.6.24@o2ib6) Sep 16 17:02:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 492a9b7d-3acb-6ef2-64e8-cfe3016d1d5c (at 10.8.7.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edf0c00, cur 1537142578 expire 1537142428 last 1537142351 Sep 16 17:04:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f4ecdfe-7c11-3195-9532-fcb464e7baba (at 10.8.2.4@o2ib6) in 202 seconds. I think it's dead, and I am evicting it. exp ffff8806d99f2400, cur 1537142654 expire 1537142504 last 1537142452 Sep 16 17:14:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 69bc8ae0-d560-1e7e-c335-63319ac6c4e2 (at 10.9.101.33@o2ib4) Sep 16 17:14:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 17:20:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 21e1d944-6ada-a5a5-2acf-1bddf182b25e (at 10.8.6.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea5000, cur 1537143634 expire 1537143484 last 1537143407 Sep 16 17:21:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 66367ed6-6ef8-36b5-068a-c130922fe990 (at 10.8.3.13@o2ib6) in 206 seconds. I think it's dead, and I am evicting it. exp ffff880ab3ef6000, cur 1537143710 expire 1537143560 last 1537143504 Sep 16 17:26:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to feeee0a9-94d1-55ba-1a7d-affde36dfe4f (at 10.8.2.4@o2ib6) Sep 16 17:26:49 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 17:34:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 101f6f96-1138-7728-6d64-22b094cb0973 (at 10.8.6.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fbafe6c00, cur 1537144486 expire 1537144336 last 1537144259 Sep 16 17:43:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec1661f5-1bf1-dafd-a8c8-d6b2df91725b (at 10.9.105.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f96a50000, cur 1537145036 expire 1537144886 last 1537144809 Sep 16 17:47:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 66367ed6-6ef8-36b5-068a-c130922fe990 (at 10.8.3.13@o2ib6) Sep 16 17:49:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dbbde0f5-0421-507a-5b2e-070c7b9ba1d8 (at 10.9.101.35@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202af79000, cur 1537145346 expire 1537145196 last 1537145119 Sep 16 17:50:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e11c5608-046e-951d-1d45-f86ff3fdb5c2 (at 10.8.6.20@o2ib6) Sep 16 17:58:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0fa2f502-c4c2-08e0-12e0-2e7c9e127f8f (at 10.9.104.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092ea03000, cur 1537145925 expire 1537145775 last 1537145698 Sep 16 18:05:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to caf800bb-b552-335d-d907-fb72a3dffd79 (at 10.9.105.23@o2ib4) Sep 16 18:05:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0e59c78-4b69-f414-027b-6a4a946291c3 (at 10.8.23.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fe25400, cur 1537146328 expire 1537146178 last 1537146101 Sep 16 18:09:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1db58f68-7c4f-2629-6fce-82554b89fc05 (at 10.8.6.9@o2ib6) Sep 16 18:09:32 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 18:09:32 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 16 18:13:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e35b3edf-e052-c573-10e3-b7303fbe865a (at 10.9.107.33@o2ib4) Sep 16 18:13:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 18:14:35 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 18:14:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 073d085a-032d-f5cd-8547-670eea44cd85 (at 10.9.104.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ce37c8400, cur 1537146893 expire 1537146743 last 1537146666 Sep 16 18:21:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0fa2f502-c4c2-08e0-12e0-2e7c9e127f8f (at 10.9.104.25@o2ib4) Sep 16 18:22:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 567f208c-fb25-d1d1-9784-9ac2d19be4d8 (at 10.8.3.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000283dc00, cur 1537147327 expire 1537147177 last 1537147100 Sep 16 18:22:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 18:24:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e0e59c78-4b69-f414-027b-6a4a946291c3 (at 10.8.23.12@o2ib6) Sep 16 18:25:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5431d30e-8d11-9511-8427-ded0c1e70da4 (at 10.8.4.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b86ff5c00, cur 1537147533 expire 1537147383 last 1537147306 Sep 16 18:31:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a96ae6b2-7b2b-728e-17e0-eaf40405df4a (at 10.8.3.19@o2ib6) Sep 16 18:42:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ef27a59f-0454-6a00-33a2-aaddfb64befe (at 10.8.4.6@o2ib6) Sep 16 18:42:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 18:59:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 15c9cdf5-6009-8f16-bc4d-397514a12548 (at 10.8.1.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fbc5ba800, cur 1537149572 expire 1537149422 last 1537149345 Sep 16 19:03:52 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 19:17:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48814a02-cad6-c053-6182-53ecfe989242 (at 10.8.22.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab430e000, cur 1537150635 expire 1537150485 last 1537150408 Sep 16 19:17:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 19:18:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb085d51-ac6f-476f-7712-352350763f74 (at 10.8.17.26@o2ib6) in 190 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e43800, cur 1537150711 expire 1537150561 last 1537150521 Sep 16 19:23:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 17b6e27b-ecd6-64c5-3e5e-45a542de047a (at 10.8.17.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9442c00, cur 1537151023 expire 1537150873 last 1537150796 Sep 16 19:25:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48814a02-cad6-c053-6182-53ecfe989242 (at 10.8.22.9@o2ib6) Sep 16 19:29:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client edfffe96-0bfd-9484-008a-07e1503a9ae4 (at 10.9.106.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e1ff13000, cur 1537151364 expire 1537151214 last 1537151137 Sep 16 19:30:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e176da3-3dc9-4a94-4d48-2b46e66ab953 (at 10.8.11.22@o2ib6) in 152 seconds. I think it's dead, and I am evicting it. exp ffff8805069ac800, cur 1537151440 expire 1537151290 last 1537151288 Sep 16 19:31:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5da164a7-4644-327d-4698-8a8c23eb538d (at 10.8.20.5@o2ib6) in 186 seconds. I think it's dead, and I am evicting it. exp ffff8800350ae000, cur 1537151516 expire 1537151366 last 1537151330 Sep 16 19:32:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c503ffb8-477e-6efc-7599-c72f5d720a65 (at 10.8.1.3@o2ib6) Sep 16 19:41:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 17b6e27b-ecd6-64c5-3e5e-45a542de047a (at 10.8.17.17@o2ib6) Sep 16 19:41:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 19:44:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2123655a-df1e-60b9-c89a-5bd65f75e522 (at 10.8.6.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880020d07c00, cur 1537152276 expire 1537152126 last 1537152049 Sep 16 19:52:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 56812f93-365a-2933-a83b-e4c779eb23da (at 10.9.106.37@o2ib4) Sep 16 20:05:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 16 20:05:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 20:10:12 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 16 20:11:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98fb3a7c-0175-768f-18d3-557aaafd2e9f (at 10.8.18.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc6c000, cur 1537153881 expire 1537153731 last 1537153654 Sep 16 20:16:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b1ab462c-3e89-0093-db3a-69918591ae2a (at 10.8.26.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8ca9a800, cur 1537154207 expire 1537154057 last 1537153980 Sep 16 20:19:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e69305dd-c324-76b2-02a0-2ddef3efb42b (at 10.8.29.8@o2ib6) Sep 16 20:19:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 20:21:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bdc2c3b-ce97-9b8a-0e3c-cd6cbfde37d9 (at 10.8.29.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005c335000, cur 1537154503 expire 1537154353 last 1537154276 Sep 16 20:22:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 92c1ca93-68ed-d713-b841-a6e421606f1a (at 10.8.26.1@o2ib6) in 178 seconds. I think it's dead, and I am evicting it. exp ffff88101be65400, cur 1537154579 expire 1537154429 last 1537154401 Sep 16 20:32:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 92c1ca93-68ed-d713-b841-a6e421606f1a (at 10.8.26.1@o2ib6) Sep 16 20:41:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cf02cdcd-783c-5bdb-99d6-40333986841c (at 10.8.2.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c27d7000, cur 1537155702 expire 1537155552 last 1537155475 Sep 16 20:41:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 20:46:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e041b8c9-dace-8861-6540-6a7fcc684186 (at 10.8.20.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef2879000, cur 1537155960 expire 1537155810 last 1537155733 Sep 16 20:50:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5be35293-3690-ba71-55ac-1eda538771b5 (at 10.8.2.3@o2ib6) Sep 16 20:50:57 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 21:10:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e041b8c9-dace-8861-6540-6a7fcc684186 (at 10.8.20.8@o2ib6) Sep 16 21:11:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0b2af4cf-18f5-f51c-78ec-db99f83a8d03 (at 10.8.21.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2c8800, cur 1537157476 expire 1537157326 last 1537157249 Sep 16 21:12:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb16a738-c5e8-0837-ac83-a869ecfc05ce (at 10.8.25.23@o2ib6) in 206 seconds. I think it's dead, and I am evicting it. exp ffff880f6cc6f400, cur 1537157552 expire 1537157402 last 1537157346 Sep 16 21:21:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 11017b5b-e1a7-5637-2fc6-1409c44e6295 (at 10.9.106.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880995a91800, cur 1537158098 expire 1537157948 last 1537157871 Sep 16 21:39:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 04298e10-7b7f-732f-23e3-2e437533d858 (at 10.8.13.26@o2ib6) Sep 16 21:44:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c2b49d28-e1d0-1e18-44c7-896c0341fb6f (at 10.9.106.42@o2ib4) Sep 16 21:44:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 21:45:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9bf10f6-a85d-06a7-bc4f-d4d24cba3c77 (at 10.8.7.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880caff20400, cur 1537159513 expire 1537159363 last 1537159286 Sep 16 21:45:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 21:50:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7949fe81-96a2-c1a3-8196-bb7238be477b (at 10.9.106.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881868a0c000, cur 1537159826 expire 1537159676 last 1537159599 Sep 16 21:50:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb16a738-c5e8-0837-ac83-a869ecfc05ce (at 10.8.25.23@o2ib6) Sep 16 22:01:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33f52eb6-932e-6608-2600-0a22e10145a0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d6e214800, cur 1537160489 expire 1537160339 last 1537160262 Sep 16 22:01:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 16 22:05:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d6d35e02-13a3-f31c-0fb2-78303e748e50 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2d635800, cur 1537160742 expire 1537160592 last 1537160515 Sep 16 22:08:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1659535f-dbe8-6dac-b1c2-ab181226e767 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015ca2000, cur 1537160938 expire 1537160788 last 1537160711 Sep 16 22:12:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a26ceda1-c107-d798-05bb-2dce6b36fbe8 (at 10.9.106.46@o2ib4) Sep 16 22:12:59 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 22:20:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d056d80d-436e-5c22-cda6-50ef052dee5a (at 10.8.31.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810392a7c00, cur 1537161607 expire 1537161457 last 1537161380 Sep 16 22:21:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 31bbf24d-6b9d-fdd1-3202-e68d68eae126 (at 10.8.11.27@o2ib6) in 199 seconds. I think it's dead, and I am evicting it. exp ffff8810392f8800, cur 1537161683 expire 1537161533 last 1537161484 Sep 16 22:22:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 390d8b58-a6e1-6466-d146-2c2783b0cd00 (at 10.9.102.20@o2ib4) in 223 seconds. I think it's dead, and I am evicting it. exp ffff882005c36800, cur 1537161759 expire 1537161609 last 1537161536 Sep 16 22:31:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 16 22:31:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 22:32:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b93fea4-a7bc-c1fe-fe94-9538e6b39ab4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fddf49800, cur 1537162354 expire 1537162204 last 1537162127 Sep 16 22:43:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 390d8b58-a6e1-6466-d146-2c2783b0cd00 (at 10.9.102.20@o2ib4) Sep 16 22:43:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 22:45:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7dad927-25d0-29df-854e-71e6e383f7c4 (at 10.8.1.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808552a3000, cur 1537163124 expire 1537162974 last 1537162897 Sep 16 22:46:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d93d6c71-18fe-592f-763c-08f957139025 (at 10.8.21.22@o2ib6) in 173 seconds. I think it's dead, and I am evicting it. exp ffff880a45b75800, cur 1537163200 expire 1537163050 last 1537163027 Sep 16 22:53:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47fbb9df-d3eb-ba37-c069-782ba5c2e44e (at 10.8.21.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3bd400, cur 1537163604 expire 1537163454 last 1537163377 Sep 16 22:59:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 159d1312-64d9-d2d5-17b0-6d77a4d1e014 (at 10.8.17.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272373400, cur 1537163983 expire 1537163833 last 1537163756 Sep 16 23:07:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5796daa5-2345-f383-9c22-e0906ac175be (at 10.9.106.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001d9dc00, cur 1537164420 expire 1537164270 last 1537164193 Sep 16 23:08:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5addab0a-5b46-9af1-6900-413b6de23c9e (at 10.8.18.35@o2ib6) in 178 seconds. I think it's dead, and I am evicting it. exp ffff88001db44000, cur 1537164496 expire 1537164346 last 1537164318 Sep 16 23:08:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 16 23:08:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 23:12:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 16 23:12:03 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 16 23:13:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79b9a04d-6023-4766-dd30-e7fe4f127ad8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802a08b0000, cur 1537164784 expire 1537164634 last 1537164557 Sep 16 23:16:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 16 23:16:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 16 23:16:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0aa4f85e-78af-7895-40c4-fe9c3e0f45be (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a936ec00, cur 1537164975 expire 1537164825 last 1537164748 Sep 16 23:21:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5796daa5-2345-f383-9c22-e0906ac175be (at 10.9.106.12@o2ib4) Sep 16 23:21:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 00:02:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7948651-5edf-3b76-d5ce-a41f91cab8cb (at 10.8.13.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880deee3c400, cur 1537167736 expire 1537167586 last 1537167509 Sep 17 00:19:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 167059a8-887d-f282-1560-0fbcdd188dab (at 10.8.13.23@o2ib6) Sep 17 00:34:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d3a741b-db7e-52ff-0eb8-5211f7d54d65 (at 10.9.114.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881865b57000, cur 1537169699 expire 1537169549 last 1537169472 Sep 17 00:48:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8c330c53-ee1a-5974-2f83-1ba3aa66ef39 (at 10.8.21.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe4448000, cur 1537170529 expire 1537170379 last 1537170302 Sep 17 00:49:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 507e1289-cf91-2386-05ae-b29860d205c8 (at 10.9.114.9@o2ib4) Sep 17 00:56:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 00:56:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e26ac576-2d9b-4238-8134-4df5ffa9081f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807444ea800, cur 1537171000 expire 1537170850 last 1537170773 Sep 17 01:07:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 01:07:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c78de01-ef7d-e6aa-fee3-c8208f0acfdb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3aff000, cur 1537171679 expire 1537171529 last 1537171452 Sep 17 01:08:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 430f70a6-735b-dec3-13cf-4f919b0b9772 (at 10.8.9.9@o2ib6) Sep 17 01:09:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a041f702-6fc2-9640-0c95-24c5d06e99b9 (at 10.8.9.9@o2ib6) in 156 seconds. I think it's dead, and I am evicting it. exp ffff8808e7092800, cur 1537171755 expire 1537171605 last 1537171599 Sep 17 01:09:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8c330c53-ee1a-5974-2f83-1ba3aa66ef39 (at 10.8.21.25@o2ib6) Sep 17 01:26:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b13febcb-aaaa-bc4c-98ad-554c9e9cc5ad (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803352b5000, cur 1537172800 expire 1537172650 last 1537172573 Sep 17 01:35:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 01:45:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b2e975b7-c2d4-1df3-3447-cadcc4ef3c06 (at 10.8.31.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810392fd800, cur 1537173956 expire 1537173806 last 1537173729 Sep 17 02:04:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b2e975b7-c2d4-1df3-3447-cadcc4ef3c06 (at 10.8.31.1@o2ib6) Sep 17 02:06:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f1ee5ce6-514b-2d4c-24b0-0ac8f3361042 (at 10.9.105.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b723c00, cur 1537175168 expire 1537175018 last 1537174941 Sep 17 02:14:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6d0a265-c3b8-356c-ef4d-cf48f0660d6f (at 10.8.26.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095ae06c00, cur 1537175696 expire 1537175546 last 1537175469 Sep 17 02:21:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 02:22:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a36682a3-446f-dc88-d20b-68d2c7b77bb5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880039fe7c00, cur 1537176128 expire 1537175978 last 1537175901 Sep 17 02:25:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client adc8e883-b26a-eed8-72ed-a356ccd8af00 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88900800, cur 1537176352 expire 1537176202 last 1537176125 Sep 17 02:28:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f1ee5ce6-514b-2d4c-24b0-0ac8f3361042 (at 10.9.105.4@o2ib4) Sep 17 02:29:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 02:33:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 02:33:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74b0334e-97ea-c02e-6260-309c07d230f6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bc6359400, cur 1537176807 expire 1537176657 last 1537176580 Sep 17 02:33:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b60f6c6-8090-7a5d-9f8c-5e8264218b67 (at 10.8.26.36@o2ib6) Sep 17 02:52:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 07989a55-c77b-b3a1-3c2e-bd7f0559cf45 (at 10.8.28.8@o2ib6) Sep 17 03:04:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ad32a28-c1bb-d588-df17-32e18e07516c (at 10.8.18.12@o2ib6) Sep 17 03:05:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f71f02e-c01f-0eab-bc6c-403bf5ff9360 (at 10.8.18.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806452e0c00, cur 1537178751 expire 1537178601 last 1537178524 Sep 17 03:05:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 03:10:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e7f0c4c0-757e-2d84-6f1c-f68c4d15d371 (at 10.8.27.15@o2ib6) Sep 17 03:10:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52feef0a-6e86-fd54-5fbc-f9293ab4f5a4 (at 10.8.27.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d10a52000, cur 1537179057 expire 1537178907 last 1537178830 Sep 17 03:15:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b4c1c7f2-cd69-7f32-f38c-a59483ececee (at 10.8.27.14@o2ib6) Sep 17 03:18:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cabd0f3b-fbd1-8356-f982-e531cc775961 (at 10.8.22.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881030655000, cur 1537179530 expire 1537179380 last 1537179303 Sep 17 03:18:50 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 17 03:27:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cabd0f3b-fbd1-8356-f982-e531cc775961 (at 10.8.22.31@o2ib6) Sep 17 03:31:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a31135bd-8bcc-7242-cf2c-3dc969b0afda (at 10.8.22.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103da72800, cur 1537180285 expire 1537180135 last 1537180058 Sep 17 03:31:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 03:33:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) Sep 17 03:36:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e8264892-03bc-53cd-683b-0cc28c0ce752 (at 10.9.104.37@o2ib4) Sep 17 03:37:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a660eb9-2bff-4e24-6e63-f94431a38a43 (at 10.8.4.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802955b6000, cur 1537180664 expire 1537180514 last 1537180437 Sep 17 03:37:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ed43e6f1-f5cf-8476-55ac-9f8ea047e06d (at 10.8.25.7@o2ib6) Sep 17 03:39:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a31135bd-8bcc-7242-cf2c-3dc969b0afda (at 10.8.22.6@o2ib6) Sep 17 03:44:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ef27a59f-0454-6a00-33a2-aaddfb64befe (at 10.8.4.6@o2ib6) Sep 17 04:05:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882002f7d400, cur 1537182343 expire 1537182193 last 1537182116 Sep 17 04:05:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 04:36:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Sep 17 04:37:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9af3ecf4-9cfd-0f66-b032-e5346067a609 (at 10.8.12.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe5a5b000, cur 1537184232 expire 1537184082 last 1537184005 Sep 17 05:01:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9af3ecf4-9cfd-0f66-b032-e5346067a609 (at 10.8.12.30@o2ib6) Sep 17 06:00:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 80099489-3226-75b7-a938-9c2e172b6eec (at 10.8.26.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b00686c00, cur 1537189231 expire 1537189081 last 1537189004 Sep 17 06:21:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1d466413-418e-db82-5337-6a2f078692ec (at 10.8.19.2@o2ib6) Sep 17 06:23:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1d466413-418e-db82-5337-6a2f078692ec (at 10.8.19.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e72c57000, cur 1537190597 expire 1537190447 last 1537190370 Sep 17 06:28:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80099489-3226-75b7-a938-9c2e172b6eec (at 10.8.26.15@o2ib6) Sep 17 06:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bd5371b7-f832-f4e8-47e8-8189f711d649 (at 10.8.23.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102abe9400, cur 1537191511 expire 1537191361 last 1537191284 Sep 17 06:47:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bd5371b7-f832-f4e8-47e8-8189f711d649 (at 10.8.23.26@o2ib6) Sep 17 06:52:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b16da9d2-fc40-20c6-1095-e256b06b48c1 (at 10.8.11.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567bf800, cur 1537192364 expire 1537192214 last 1537192137 Sep 17 07:04:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f181bc36-12f4-d13f-cac6-685c8fca4703 (at 10.8.11.21@o2ib6) Sep 17 07:17:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b16da9d2-fc40-20c6-1095-e256b06b48c1 (at 10.8.11.26@o2ib6) Sep 17 07:25:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d4412a7a-2f1c-7c86-c893-8f2d24020066 (at 10.8.20.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880225010400, cur 1537194347 expire 1537194197 last 1537194120 Sep 17 07:25:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 07:53:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4412a7a-2f1c-7c86-c893-8f2d24020066 (at 10.8.20.22@o2ib6) Sep 17 08:31:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5158eccf-14ae-5d94-c749-4edeb98a44a2 (at 10.8.6.8@o2ib6) Sep 17 08:43:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bccf629c-900e-c4a9-171a-d70129138265 (at 10.8.14.1@o2ib6) Sep 17 08:50:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97f995f6-e443-5d59-c793-ddc9590554c0 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca1400, cur 1537199428 expire 1537199278 last 1537199201 Sep 17 09:00:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 17 09:10:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a3711652-2225-f403-251a-9e82e770218d (at 10.8.20.2@o2ib6) Sep 17 09:21:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 17 09:22:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 17 09:22:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d65d6a82-e961-8992-2e5b-f5d98119df47 (at 10.8.14.9@o2ib6) Sep 17 09:22:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4966203-761f-6dc5-9911-bf9e91bbb908 (at 10.8.13.24@o2ib6) Sep 17 09:23:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4f8ff9b5-3c9f-09af-6ef9-a813290ae6d7 (at 10.8.14.8@o2ib6) Sep 17 09:24:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4a9b9ea1-3786-1d90-2c15-70a68834d78c (at 10.8.3.1@o2ib6) Sep 17 09:24:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 09:24:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 17 09:25:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e77cf706-7227-c9f7-af51-d5ef8bbca1cf (at 10.8.17.23@o2ib6) Sep 17 09:30:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 17 09:34:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f735048-8e18-63f4-ba31-ccd05dbd9e5d (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800104df800, cur 1537202070 expire 1537201920 last 1537201843 Sep 17 09:35:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 17 09:35:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 17 09:42:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 17 09:44:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ee87d2e4-a37e-347b-d9fd-b6ebb5e29d24 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053211000, cur 1537202663 expire 1537202513 last 1537202436 Sep 17 10:04:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 17 10:04:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 10:11:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0d34e2c-d599-78df-9f57-d05952425498 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880438c6fc00, cur 1537204260 expire 1537204110 last 1537204033 Sep 17 10:11:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 10:22:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3427dd05-b49e-520d-ab57-b133ca713edf (at 10.8.8.32@o2ib6) Sep 17 10:24:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42e493fa-5839-47ad-6bbd-80f855278552 (at 10.9.113.8@o2ib4) Sep 17 10:29:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8000ef19-6995-0fb5-b64b-95113c08faf4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801fc9d3800, cur 1537205367 expire 1537205217 last 1537205140 Sep 17 10:29:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 10:30:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bccf629c-900e-c4a9-171a-d70129138265 (at 10.8.14.1@o2ib6) Sep 17 12:54:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 280b6d0c-1256-aed6-77fc-0666da0322f9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5fcf0c00, cur 1537214067 expire 1537213917 last 1537213840 Sep 17 12:54:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 12:58:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 12:59:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71f79049-802d-3119-f8d8-d6a3e131e2c9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e6ba49c00, cur 1537214379 expire 1537214229 last 1537214152 Sep 17 13:11:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 13:12:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 18a5de3f-9b44-cb83-67b7-077817afc9a7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c62f8c00, cur 1537215125 expire 1537214975 last 1537214898 Sep 17 14:09:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d112f4b8-8774-9ca1-4444-b197b6ab5508 (at 10.9.107.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f59a46400, cur 1537218575 expire 1537218425 last 1537218348 Sep 17 16:46:50 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 16:56:30 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 16:58:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 16:58:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a3ceb46-e67e-7a8d-9916-55152b1efbcc (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808d6af2800, cur 1537228739 expire 1537228589 last 1537228512 Sep 17 17:02:19 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.245@o2ib5 version 12/12 incarnation 1536017134150867/1537228912189410 Sep 17 17:02:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fccddc08-d568-f4b4-6a44-f4e21b04be63 (at 10.0.2.245@o2ib5) Sep 17 17:03:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 17:03:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2dd17812-92ba-676e-abd1-960496b7b488 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc6d34400, cur 1537229021 expire 1537228871 last 1537228794 Sep 17 17:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c9da9378-ead2-7fa4-c753-767f61fd75de (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b99267800, cur 1537229342 expire 1537229192 last 1537229115 Sep 17 17:09:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 17 17:18:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 17:20:40 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 17:23:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 17:23:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e96d2f1e-889e-9fe5-b6dd-794337755319 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7de6c800, cur 1537230236 expire 1537230086 last 1537230009 Sep 17 17:30:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6cb49a44-1b08-1885-95bf-56bf4ce38a9e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880017fbb800, cur 1537230608 expire 1537230458 last 1537230381 Sep 17 17:38:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 17:43:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 17:44:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e64ed17-2050-7ea8-653c-b9306669fb30 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003fbc8800, cur 1537231458 expire 1537231308 last 1537231231 Sep 17 17:51:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 17:52:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b38d5a7d-a091-499a-b266-6634a9276290 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880038efe400, cur 1537231957 expire 1537231807 last 1537231730 Sep 17 17:55:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 17:56:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b969ab9-a5df-b167-85b4-b5ec16b216e9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f67198000, cur 1537232174 expire 1537232024 last 1537231947 Sep 17 17:57:55 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 18:10:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 18:11:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93f2fc11-077a-cdb9-795e-a22b14d4be21 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f76f44000, cur 1537233080 expire 1537232930 last 1537232853 Sep 17 18:14:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9487d624-73b2-3c19-bebc-c7865b0a5ce5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880385273400, cur 1537233299 expire 1537233149 last 1537233072 Sep 17 18:23:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 18:31:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e8f388f-c10b-6217-3d48-939e227fe10f (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d39bbf000, cur 1537234289 expire 1537234139 last 1537234062 Sep 17 18:34:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 17 18:49:00 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 18:49:00 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 17 18:49:04 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 18:56:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9d84c474-75a0-9041-3876-4df8bb862feb (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88048a9efc00, cur 1537235794 expire 1537235644 last 1537235567 Sep 17 19:07:11 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 19:20:32 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 19:28:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e443e22-214e-df90-07fc-c40496ad920c (at 10.8.1.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff6157400, cur 1537237690 expire 1537237540 last 1537237463 Sep 17 19:35:37 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 19:39:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d77d516-869c-a702-dff0-3753befd95cb (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880463a32000, cur 1537238390 expire 1537238240 last 1537238163 Sep 17 19:48:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 17 19:48:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c09c47e5-a142-864b-1012-ed0029e782a1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005b62c800, cur 1537238913 expire 1537238763 last 1537238686 Sep 17 19:48:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 17 20:03:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2e443e22-214e-df90-07fc-c40496ad920c (at 10.8.1.26@o2ib6) Sep 17 20:07:55 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 17 20:28:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2479713c-d92a-9a4c-31db-188d0ee65bc5 (at 10.8.11.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102abe9c00, cur 1537241299 expire 1537241149 last 1537241072 Sep 17 20:29:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fbf98194-b41b-5b2e-555d-200bf97053ef (at 10.12.4.55@o2ib) in 208 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1a000, cur 1537241375 expire 1537241225 last 1537241167 Sep 17 20:32:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 003bfd88-919c-3745-858b-eadb4f8ad9d1 (at 10.12.4.55@o2ib) Sep 17 20:52:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ca21d097-a095-ce1a-3050-8700484deba8 (at 10.12.4.43@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101cff0c00, cur 1537242745 expire 1537242595 last 1537242518 Sep 17 22:43:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04e0325b-0638-2ea2-5d3f-1bd2cbf4f248 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88953400, cur 1537249392 expire 1537249242 last 1537249165 Sep 17 23:51:53 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 00:34:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 25c04b3d-1298-d2d3-0ef1-dd200490a8cd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88015bc8f400, cur 1537256041 expire 1537255891 last 1537255814 Sep 18 00:35:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 00:37:45 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 00:39:06 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 00:39:07 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 01:30:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d84a6811-d519-52d8-9f07-4ff8ac852bd5 (at 10.8.1.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800238ccc00, cur 1537259431 expire 1537259281 last 1537259204 Sep 18 04:16:23 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 04:16:23 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 18 04:20:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33afb353-9052-7b46-d83b-6d9440cf8df0 (at 10.8.10.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88053d6c8400, cur 1537269634 expire 1537269484 last 1537269407 Sep 18 04:30:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8d15b20b-cce0-3a08-fe85-16ffc43b17df (at 10.8.10.1@o2ib6) Sep 18 05:55:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9c67b9e-5ae3-8fb4-77ba-ac1bc34b711e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4a35bc00, cur 1537275317 expire 1537275167 last 1537275090 Sep 18 05:55:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2aa107f8-781e-d9fa-9171-25d77f3a571e (at 10.8.11.22@o2ib6) Sep 18 06:08:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e711427-a20d-a246-1d68-feb4f018da9f (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fceac0c00, cur 1537276080 expire 1537275930 last 1537275853 Sep 18 06:28:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 18 09:22:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) Sep 18 09:28:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 06e545ae-a703-1875-403e-d129c47129aa (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beba43800, cur 1537288080 expire 1537287930 last 1537287853 Sep 18 09:45:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 18 09:46:15 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 09:46:15 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 18 09:54:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 09:58:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bc6402d-b378-6e1a-05ed-3929231ff59b (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5cf02c00, cur 1537289894 expire 1537289744 last 1537289667 Sep 18 10:04:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef9a4f34-14c5-cce1-34fc-70ef87f15992 (at 10.8.6.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880551973c00, cur 1537290279 expire 1537290129 last 1537290052 Sep 18 10:06:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 18 10:07:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 10:12:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec62c46d-fe59-ab04-0e95-c067c3f4001a (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806345ce000, cur 1537290725 expire 1537290575 last 1537290498 Sep 18 10:13:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 10:19:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 10:19:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6500a99a-92aa-557c-ce2f-9f5285194796 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040036400, cur 1537291194 expire 1537291044 last 1537290967 Sep 18 10:31:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f2874060-6bfb-c6c5-c8a8-60d8ca27c3be (at 10.8.6.7@o2ib6) Sep 18 10:36:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ab38002-3b40-1d5b-0d7a-d6109c8d1063 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802a08b2400, cur 1537292166 expire 1537292016 last 1537291939 Sep 18 10:36:19 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 10:36:19 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 18 10:36:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 10:51:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 535751ae-138c-38e8-8aa4-7440b14846c4 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c5ca6c400, cur 1537293109 expire 1537292959 last 1537292882 Sep 18 11:07:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 18 11:13:15 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 11:14:46 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 11:15:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 11:23:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f679004e-744a-b80b-c299-540eb2e2dd70 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880503104c00, cur 1537295032 expire 1537294882 last 1537294805 Sep 18 11:23:52 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 18 11:23:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 11:33:44 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 11:33:44 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 18 11:40:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 18 11:49:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 32b2dd76-c01a-a164-d2eb-afd7ffd8c8d8 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2108000, cur 1537296588 expire 1537296438 last 1537296361 Sep 18 11:50:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 11:57:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 852129b6-6141-ae5a-d431-07129369e0bb (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c40fe6c00, cur 1537297057 expire 1537296907 last 1537296830 Sep 18 11:58:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7c68cd06-d922-b25c-f854-630170a4cb9d (at 10.8.4.15@o2ib6) in 183 seconds. I think it's dead, and I am evicting it. exp ffff8806be2e5400, cur 1537297133 expire 1537296983 last 1537296950 Sep 18 11:59:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 12:06:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 12:10:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 12:11:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb465c7a-0291-cfad-ad7b-6b514c5984be (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b8b434800, cur 1537297896 expire 1537297746 last 1537297669 Sep 18 12:21:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 18 12:28:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 920d9047-0ed1-53f3-ae76-403d6cccce2d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807c7f96c00, cur 1537298909 expire 1537298759 last 1537298682 Sep 18 12:28:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 12:29:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a7ff2b86-d53c-fe29-8910-e94aae213c33 (at 10.8.4.15@o2ib6) in 216 seconds. I think it's dead, and I am evicting it. exp ffff880ef8784800, cur 1537298985 expire 1537298835 last 1537298769 Sep 18 12:31:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 12:34:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 829b93b5-8791-c5cf-ee7a-a524e4819240 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880420555c00, cur 1537299245 expire 1537299095 last 1537299018 Sep 18 12:44:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 12:45:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 999890dd-e435-0c11-4983-1185bbf74a4d (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4fb13c00, cur 1537299955 expire 1537299805 last 1537299728 Sep 18 12:51:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c1efe321-15a6-d3fd-e03a-6ac210a58df9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88015a89c000, cur 1537300260 expire 1537300110 last 1537300033 Sep 18 13:00:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 13:03:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be0c2e6f-224b-0809-e7ee-700df8c8a14b (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806d99f5400, cur 1537300992 expire 1537300842 last 1537300765 Sep 18 13:03:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 13:09:40 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 13:12:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d408b072-2a94-f635-b851-78af7886552c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880530fdc800, cur 1537301563 expire 1537301413 last 1537301336 Sep 18 13:15:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8a38debe-c253-ad32-64a7-e4d8fe52f711 (at 10.9.105.32@o2ib4) Sep 18 13:21:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 13:28:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0128da3-7b3b-3cb3-5e12-cb2c90bb988f (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fceac3800, cur 1537302486 expire 1537302336 last 1537302259 Sep 18 13:29:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ddd563e7-3399-7871-ad8a-b10cc4ca3aa1 (at 10.8.17.26@o2ib6) in 225 seconds. I think it's dead, and I am evicting it. exp ffff880074f67800, cur 1537302562 expire 1537302412 last 1537302337 Sep 18 13:29:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Sep 18 13:30:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 13:36:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0113a85-0352-430e-d100-40b7b4e85458 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a12a88c00, cur 1537303017 expire 1537302867 last 1537302790 Sep 18 13:46:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 13:50:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 13:51:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f3b93483-b759-db45-ceb0-aa6fc34d9528 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800246d9c00, cur 1537303867 expire 1537303717 last 1537303640 Sep 18 13:53:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 13:54:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ded1a4d0-3de3-a7ec-f9e1-a0b18fc041d9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880025af2000, cur 1537304056 expire 1537303906 last 1537303829 Sep 18 14:03:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30aafa4d-0213-007a-b12a-3251e2ad77f0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1a000, cur 1537304588 expire 1537304438 last 1537304361 Sep 18 14:03:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5158eccf-14ae-5d94-c749-4edeb98a44a2 (at 10.8.6.8@o2ib6) Sep 18 14:09:36 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 14:11:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fcfa2a39-06c7-bc67-fb76-85aa1db4e549 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880caff24400, cur 1537305091 expire 1537304941 last 1537304864 Sep 18 14:11:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 14:11:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 14:15:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 14:16:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4875ee55-1e00-b52c-9b1f-76625da04538 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802f82d7c00, cur 1537305369 expire 1537305219 last 1537305142 Sep 18 14:18:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 14:19:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e400572-443b-532f-b32a-45d87d544b35 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff3c79c00, cur 1537305567 expire 1537305417 last 1537305340 Sep 18 14:19:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 14:24:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a61f143a-b347-f388-1921-9b6e1c9aa912 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5285a000, cur 1537305882 expire 1537305732 last 1537305655 Sep 18 14:25:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 14:45:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 14:50:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98e8a79c-4b29-6877-e85c-ad65fb9d921a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4cc34800, cur 1537307458 expire 1537307308 last 1537307231 Sep 18 15:00:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 15:00:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:05:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:05:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 06b277c2-c7b1-415f-197a-48c266b66a09 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808ecefb000, cur 1537308333 expire 1537308183 last 1537308106 Sep 18 15:05:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 15:08:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 478b4a93-6ee2-be9d-66ee-726bf2d1c4d0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a82963000, cur 1537308542 expire 1537308392 last 1537308315 Sep 18 15:11:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:12:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b04caa79-c757-e54d-36ef-eb5e2fbe94a0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a089db400, cur 1537308725 expire 1537308575 last 1537308498 Sep 18 15:14:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:15:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 80282725-538d-88eb-61ab-1546dab0a3da (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805b8139800, cur 1537308910 expire 1537308760 last 1537308683 Sep 18 15:18:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:18:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c0f9e41-23e0-b32a-6dd2-15462c397a49 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee9ff4c00, cur 1537309122 expire 1537308972 last 1537308895 Sep 18 15:19:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 15:19:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ff04feb2-8ec4-216d-ef94-a603cb4be3ca (at 10.8.20.19@o2ib6) in 220 seconds. I think it's dead, and I am evicting it. exp ffff880b29bfb000, cur 1537309198 expire 1537309048 last 1537308978 Sep 18 15:25:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7ce67186-fc76-9d5f-e736-a2320e9b3fef (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd64b800, cur 1537309544 expire 1537309394 last 1537309317 Sep 18 15:27:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dbf7a426-22a1-a952-d947-2a83310c04f7 (at 10.8.17.26@o2ib6) in 207 seconds. I think it's dead, and I am evicting it. exp ffff880d169c8000, cur 1537309620 expire 1537309470 last 1537309413 Sep 18 15:27:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:28:39 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 15:32:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 096b12e9-dad2-58a7-942d-f252118f99c3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880046e2b400, cur 1537309956 expire 1537309806 last 1537309729 Sep 18 15:33:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 15:38:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9b380b9-8e32-c750-ebaf-d6c56d5039fd (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881008641800, cur 1537310287 expire 1537310137 last 1537310060 Sep 18 15:38:46 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 15:42:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d66ccf59-be80-3722-7b68-2e6c575b2f44 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9a400, cur 1537310559 expire 1537310409 last 1537310332 Sep 18 15:43:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:43:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 18 15:54:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f89c9ca-f485-174e-6078-5ecc4ada4831 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d598ac400, cur 1537311267 expire 1537311117 last 1537311040 Sep 18 15:54:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 15:55:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 15:55:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 16:17:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ae2b5d9-83e3-a025-5196-0729425921e8 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880585ab5800, cur 1537312660 expire 1537312510 last 1537312433 Sep 18 16:17:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 16:18:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5fb42f44-6d20-1500-290f-13e593b5bb24 (at 10.8.6.15@o2ib6) Sep 18 16:22:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 357a482e-c4fc-c410-ee67-77160a26dcd2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015e4b000, cur 1537312934 expire 1537312784 last 1537312707 Sep 18 16:22:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 16:22:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 16:26:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1805d7f9-c3c7-b23e-389f-ae0e2d51d161 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800670c6000, cur 1537313219 expire 1537313069 last 1537312992 Sep 18 16:36:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 16:37:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 16:41:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 16:41:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c1222b5c-078f-3c02-39ee-b2841d425e2d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be07800, cur 1537314083 expire 1537313933 last 1537313856 Sep 18 16:41:23 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 18 16:43:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 16:47:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 18 16:56:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 18 16:57:39 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 17:10:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 17:10:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6529b3ef-cd5c-7de5-1722-6b09bd7db547 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f03e19000, cur 1537315847 expire 1537315697 last 1537315620 Sep 18 17:10:47 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 18 17:12:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a117be7e-20f6-8c99-4d04-7f1f0cbf7b22 (at 10.8.18.35@o2ib6) in 201 seconds. I think it's dead, and I am evicting it. exp ffff880f9775d800, cur 1537315923 expire 1537315773 last 1537315722 Sep 18 17:16:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e80f9ba8-03c5-7c65-488a-50d73c516680 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee03b2000, cur 1537316162 expire 1537316012 last 1537315935 Sep 18 17:21:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 17:21:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 17:25:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe8044d8-d181-ee65-aae0-16545cf5b946 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040d11000, cur 1537316741 expire 1537316591 last 1537316514 Sep 18 17:25:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 17:37:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 17:37:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 18 17:37:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7420ac08-d876-8052-9df6-f6916ec7bd8c (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ef2418800, cur 1537317476 expire 1537317326 last 1537317249 Sep 18 17:37:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 17:51:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a91d84fc-4aaf-2259-1e96-225eda77e1a0 (at 10.9.102.8@o2ib4) Sep 18 17:51:21 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 18 17:51:33 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 17:55:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1b918d0b-f652-e470-9fb9-778875bd7635 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f19682800, cur 1537318514 expire 1537318364 last 1537318287 Sep 18 17:55:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 18:02:33 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 18:07:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2930f5e3-30fe-ae97-97b6-0c0bf8caca7d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802feea0800, cur 1537319265 expire 1537319115 last 1537319038 Sep 18 18:07:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 18:07:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 18:25:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cbf2bfa2-993c-0a43-a564-86f8b36f773a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005a0af400, cur 1537320318 expire 1537320168 last 1537320091 Sep 18 18:26:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a91d84fc-4aaf-2259-1e96-225eda77e1a0 (at 10.9.102.8@o2ib4) Sep 18 18:38:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 18:38:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 18:38:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bab8753-d3d3-93f5-d6e7-836a2fb0173c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ccbfd2000, cur 1537321137 expire 1537320987 last 1537320910 Sep 18 18:38:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 18:38:58 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 18:52:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 18:52:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 18:53:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b791e6f-3169-165f-e2e0-1a270874db6c (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c62ff400, cur 1537322012 expire 1537321862 last 1537321785 Sep 18 18:53:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 18 19:02:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 19:02:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 19:07:27 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 19:08:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4551b595-6c4b-e96d-9b2d-dfecca68f2c5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cf76c9800, cur 1537322882 expire 1537322732 last 1537322655 Sep 18 19:08:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 19:12:14 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 19:16:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 19:16:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 19:43:19 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 19:48:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 19:49:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b595bfa7-4d63-d23a-607f-338cacbed065 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e3c58400, cur 1537325362 expire 1537325212 last 1537325135 Sep 18 19:51:09 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 19:51:09 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 18 19:52:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 19:53:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 84bb5601-13af-4f37-5e40-c155691e8f3c (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88037d285c00, cur 1537325584 expire 1537325434 last 1537325357 Sep 18 19:56:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7027c055-5a6a-27c3-a022-a22ccf3de56a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804ad8e4800, cur 1537325786 expire 1537325636 last 1537325559 Sep 18 20:02:11 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 20:02:11 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 18 20:05:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 20:10:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d049ae5b-d9bd-288c-d311-b5994ef031a8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88029d91e800, cur 1537326629 expire 1537326479 last 1537326402 Sep 18 20:19:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 20:24:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd971daa-2e49-1787-95c7-5369b5ec3493 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e151e9c00, cur 1537327457 expire 1537327307 last 1537327230 Sep 18 20:32:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 20:54:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 18 21:13:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 21:14:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a6aa8f4-e58b-e523-a762-c7c73f6d9c6d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b29bf8000, cur 1537330452 expire 1537330302 last 1537330225 Sep 18 21:14:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 18 21:17:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 492774f8-912e-48e3-b035-c063d878a63d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802c4eb7400, cur 1537330658 expire 1537330508 last 1537330431 Sep 18 21:20:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 21:26:13 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 21:27:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 18 21:28:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b35a1576-a27a-49be-36ac-76cf91376e29 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b28fdbc00, cur 1537331328 expire 1537331178 last 1537331101 Sep 18 21:34:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 54189dd7-932d-6715-248c-879511376461 (at 10.12.4.45@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810302f5c00, cur 1537331695 expire 1537331545 last 1537331468 Sep 18 21:54:41 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 22:05:27 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 22:13:24 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 18 22:26:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca21d097-a095-ce1a-3050-8700484deba8 (at 10.12.4.43@o2ib) Sep 18 22:26:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 54189dd7-932d-6715-248c-879511376461 (at 10.12.4.45@o2ib) Sep 18 22:55:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 18 22:55:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a873963-927b-411a-c894-674ee7dbd8c5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a12a8cc00, cur 1537336550 expire 1537336400 last 1537336323 Sep 18 23:24:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 706e95e5-fa9a-b2ef-dd5c-7ccc855c29e2 (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1b000, cur 1537338276 expire 1537338126 last 1537338049 Sep 18 23:55:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7737b6c5-b9fd-ad2f-3fc6-e52ae308bc51 (at 10.8.14.5@o2ib6) Sep 19 00:28:02 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 00:28:03 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 00:43:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c4d38f5-23f2-9ad1-5db4-78b00f58b931 (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809dc2a6000, cur 1537343034 expire 1537342884 last 1537342807 Sep 19 01:04:33 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 01:04:33 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 01:08:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Sep 19 01:42:00 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 02:55:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d92b2175-d59a-6203-3a31-c1e39cfb1241 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880073b19c00, cur 1537350930 expire 1537350780 last 1537350703 Sep 19 02:57:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 19 02:58:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55e136a9-0b2b-e889-8aed-f5f3536f0539 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800648e9800, cur 1537351091 expire 1537350941 last 1537350864 Sep 19 03:25:33 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 03:52:39 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 03:52:39 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 19 04:02:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7892d78b-47db-43ac-3839-a7398321959c (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b1323cc00, cur 1537354959 expire 1537354809 last 1537354732 Sep 19 04:51:43 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 04:51:43 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 05:03:02 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 05:03:02 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 19 05:06:28 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 05:06:28 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 19 05:12:47 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 05:47:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 03db311d-9c45-21ef-3c73-4af98e82c8ba (at 10.8.6.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802feea2800, cur 1537361255 expire 1537361105 last 1537361028 Sep 19 05:58:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5158eccf-14ae-5d94-c749-4edeb98a44a2 (at 10.8.6.8@o2ib6) Sep 19 06:11:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a3e19c13-7c5c-8db0-f365-e3ce40a9c471 (at 10.8.6.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d288ba800, cur 1537362717 expire 1537362567 last 1537362490 Sep 19 06:38:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e5dba0a1-6752-d1db-8cce-13d7bdc15bd1 (at 10.8.6.28@o2ib6) Sep 19 08:49:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Sep 19 08:49:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 19 08:49:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Sep 19 08:50:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Sep 19 09:07:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 19 09:08:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2479713c-d92a-9a4c-31db-188d0ee65bc5 (at 10.8.11.14@o2ib6) Sep 19 09:26:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Sep 19 09:28:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae8a2564-1dc5-e931-8c6e-be97a7b0ae5d (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f6800, cur 1537374539 expire 1537374389 last 1537374312 Sep 19 09:29:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 39d891b9-c4f4-dbea-4a6b-207a63490d00 (at 10.9.107.22@o2ib4) Sep 19 09:38:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 19 09:42:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c52b8335-593d-173d-8555-5df5d409ceb0 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807f58dac00, cur 1537375362 expire 1537375212 last 1537375135 Sep 19 09:44:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 19 09:58:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9742c80-2b61-51b2-bcf6-97287427a1c8 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc6d32c00, cur 1537376320 expire 1537376170 last 1537376093 Sep 19 09:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 19 10:25:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 19 10:29:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 19 10:30:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 496daade-49e2-851d-54be-0debe715abf0 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039232c00, cur 1537378207 expire 1537378057 last 1537377980 Sep 19 10:38:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 585d8ff8-4575-4ba4-5af4-02aa57043521 (at 10.8.3.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063427c00, cur 1537378736 expire 1537378586 last 1537378509 Sep 19 10:38:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 19 10:48:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c26648fa-7ebe-2fb6-6200-13f31b24559d (at 10.8.6.30@o2ib6) Sep 19 10:50:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 19 10:52:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14120953-97c0-d91b-57b7-83eabdcf79de (at 10.8.21.16@o2ib6) Sep 19 10:53:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 108ae030-b894-086f-f4dc-dc1932b3417f (at 10.8.6.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807f7a51800, cur 1537379581 expire 1537379431 last 1537379354 Sep 19 10:55:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6182812c-80b3-9c9e-c311-9db8295eead5 (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef45b400, cur 1537379750 expire 1537379600 last 1537379523 Sep 19 11:31:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 585d8ff8-4575-4ba4-5af4-02aa57043521 (at 10.8.3.2@o2ib6) Sep 19 11:54:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc951c34-dec9-4330-5a4b-9f82eb7df46a (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88006648f000, cur 1537383272 expire 1537383122 last 1537383045 Sep 19 11:54:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 19 11:56:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 11:57:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 308086c6-91b2-a90e-f1c3-374a2c79e302 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee056e000, cur 1537383450 expire 1537383300 last 1537383223 Sep 19 12:01:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39dd66ab-44b9-9447-dd85-f9cb7ff44775 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e38800, cur 1537383714 expire 1537383564 last 1537383487 Sep 19 12:02:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 12:06:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b91958a0-2964-4792-c542-c7d0f5b90d6d (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807562d3400, cur 1537383965 expire 1537383815 last 1537383738 Sep 19 12:06:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 12:14:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eec41b63-f8d4-7136-451e-52af4ac1a2f6 (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f52949800, cur 1537384480 expire 1537384330 last 1537384253 Sep 19 12:14:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 19 12:19:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ef02aec4-f626-b40d-d404-973e88e07efe (at 10.8.0.65@o2ib6) Sep 19 12:19:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 19 12:38:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3e6b8413-a309-e923-5342-2526cadfe821 (at 10.8.0.66@o2ib6) Sep 19 14:07:37 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 14:07:38 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 14:07:38 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 14:36:32 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 14:36:32 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 14:50:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cdcd05c3-54f5-8748-3ca6-a470bb0610fb (at 10.8.4.9@o2ib6) Sep 19 15:00:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a229ada3-99e5-45f8-0ba6-7ede88791148 (at 10.8.4.2@o2ib6) Sep 19 15:04:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97ccc11e-a6f7-ff81-6fb5-c5a860dcc208 (at 10.8.6.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014c055c00, cur 1537394688 expire 1537394538 last 1537394461 Sep 19 15:07:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Sep 19 15:11:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b020de1-e96d-8f65-4301-37ed1264617c (at 10.9.102.65@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a51800, cur 1537395080 expire 1537394930 last 1537394853 Sep 19 15:33:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6b020de1-e96d-8f65-4301-37ed1264617c (at 10.9.102.65@o2ib4) Sep 19 16:24:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5098bf3-f910-f5e4-2fac-e868feeae775 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e387400, cur 1537399448 expire 1537399298 last 1537399221 Sep 19 16:32:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 16:33:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e1b39f46-5918-fc9f-06dc-88301d32ab75 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880860fa3000, cur 1537400006 expire 1537399856 last 1537399779 Sep 19 16:41:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2cf34e5-b4a0-346d-a446-397a7bbae2aa (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d094bec00, cur 1537400476 expire 1537400326 last 1537400249 Sep 19 16:41:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 16:47:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 16:47:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eacdde6f-33b9-5c0e-5e18-cd50e13782c8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd633d000, cur 1537400872 expire 1537400722 last 1537400645 Sep 19 16:51:19 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 16:51:19 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 16:51:27 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 16:51:27 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 19 16:52:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 16:52:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99be6917-c6e3-402c-7a27-7163a9eeb5d1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039231c00, cur 1537401153 expire 1537401003 last 1537400926 Sep 19 16:58:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 733f5fd5-ce5b-def0-ca13-b99cc54ff759 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc6d33c00, cur 1537401516 expire 1537401366 last 1537401289 Sep 19 16:58:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 17:04:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a243cf45-3726-40e4-70dd-2e0256da100a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d2690000, cur 1537401858 expire 1537401708 last 1537401631 Sep 19 17:14:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 17:21:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bdaf1b2-5d98-5b12-6c44-480ab5949aa3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88070479f000, cur 1537402867 expire 1537402717 last 1537402640 Sep 19 17:22:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 17:27:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b551092-d474-801d-ebb6-18138fd08ef8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88029d8eac00, cur 1537403230 expire 1537403080 last 1537403003 Sep 19 17:35:45 oak-md1-s2 kernel: Lustre: 10530:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 17:36:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 17:40:23 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 17:40:23 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 17:40:27 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 17:41:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 17:41:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98ab9fd5-a7dd-959c-ffe1-dd1196b3e5a7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018c4bc000, cur 1537404096 expire 1537403946 last 1537403869 Sep 19 17:45:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9dd99b20-372e-8877-f9a9-d9aa61056ada (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9f800, cur 1537404320 expire 1537404170 last 1537404093 Sep 19 17:46:01 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 17:46:01 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 19 17:54:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 18:01:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 18:01:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 516cb7d6-5100-3228-bd07-4368ee6467a0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe7fa1000, cur 1537405302 expire 1537405152 last 1537405075 Sep 19 18:06:27 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 18:06:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 18:07:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a0856f0d-8e67-ea32-5302-d8f0d7239c65 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880caff5c800, cur 1537405625 expire 1537405475 last 1537405398 Sep 19 18:15:38 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 18:15:38 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Sep 19 18:22:42 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 18:22:42 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Sep 19 18:31:18 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 18:31:18 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 19 18:42:00 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 18:42:00 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 18:42:09 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 19:05:12 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 19:05:12 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 19 19:05:57 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 19:08:34 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 19:09:11 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 19:09:11 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 19 19:11:53 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 19:11:53 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 19 19:43:40 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 19:43:40 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 19 20:18:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 20:18:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55e2ef64-440e-f7cf-3700-88a1615a3fed (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806452e4800, cur 1537413531 expire 1537413381 last 1537413304 Sep 19 20:52:57 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 20:53:33 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 20:55:15 oak-md1-s2 kernel: Lustre: 3869:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 20:56:53 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 20:56:53 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 19 21:00:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 21:00:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c87e0f38-ced7-4edd-ae9d-a10a0c298f85 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f1f664800, cur 1537416041 expire 1537415891 last 1537415814 Sep 19 21:02:41 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 21:02:41 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 21:03:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 21:04:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db138e26-0014-a8da-8b0a-991502f9e7fc (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88067bedb000, cur 1537416281 expire 1537416131 last 1537416054 Sep 19 21:08:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de16894e-2707-d08f-f7f4-8831297a9b73 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880afabafc00, cur 1537416481 expire 1537416331 last 1537416254 Sep 19 21:17:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 21:21:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79e359ab-51fb-6921-7276-6ad117492f5d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880353c3e400, cur 1537417283 expire 1537417133 last 1537417056 Sep 19 21:21:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 21:25:16 oak-md1-s2 kernel: LustreError: 10671:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.0.62@o2ib4) failed to reply to blocking AST (req@ffff88150cf88c00 x1610627769919248 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881906273400/0x6c00c3ee8b8246e7 lrc: 4/0,0 mode: PR/PR res: [0x20001738c:0x119f2:0x0].0x0 bits 0x1b rrc: 5 type: IBT flags: 0x54a01400000020 nid: 10.9.0.62@o2ib4 remote: 0xb7b5e6e13c6e3bd0 expref: 23827 pid: 10671 timeout: 5702609484 lvb_type: 0 Sep 19 21:25:16 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.0.62@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Sep 19 21:30:59 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 21:30:59 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 19 21:36:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 21:36:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9cbbedc9-0b60-4583-e750-ebc8ab67478d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802f1b61c00, cur 1537418215 expire 1537418065 last 1537417988 Sep 19 21:44:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 21:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dcad6925-e82e-b85a-3c96-4b3ed0c89c50 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015e4bc00, cur 1537418689 expire 1537418539 last 1537418462 Sep 19 21:54:52 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 21:54:52 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 19 22:03:58 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 22:03:58 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 19 22:15:47 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 22:15:47 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 19 22:15:48 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 22:17:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 22:17:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a47935e5-2eda-a6b3-7907-fabc00b33451 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c875400, cur 1537420669 expire 1537420519 last 1537420442 Sep 19 22:24:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc5fbf20-b1e4-a50e-f641-537a4b9a0d66 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d18d08c00, cur 1537421099 expire 1537420949 last 1537420872 Sep 19 22:26:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 212771eb-9781-3874-237d-fb1da312e19a (at 10.8.18.35@o2ib6) in 221 seconds. I think it's dead, and I am evicting it. exp ffff880074c82400, cur 1537421175 expire 1537421025 last 1537420954 Sep 19 22:35:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Sep 19 22:38:36 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 22:38:36 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 22:42:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 22:47:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 22:47:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 405ceb31-55d3-2187-542b-fef0f152de46 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5cf00800, cur 1537422430 expire 1537422280 last 1537422203 Sep 19 22:54:41 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 22:54:41 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 22:54:44 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 22:59:34 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 23:13:02 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 23:13:02 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 19 23:24:57 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 23:24:57 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 19 23:24:58 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 19 23:24:58 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 19 23:37:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 19 23:38:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d6f434cf-4ae2-1508-cada-b2b8b655e902 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ba78c3c00, cur 1537425498 expire 1537425348 last 1537425271 Sep 20 00:31:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 20 00:32:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a121b0e-af99-3149-c4ee-fffad27d9312 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802ca315c00, cur 1537428744 expire 1537428594 last 1537428517 Sep 20 00:34:29 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 00:34:29 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 20 00:38:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6752f70d-207e-4a3d-6676-c48ead02eceb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880995a96000, cur 1537429113 expire 1537428963 last 1537428886 Sep 20 00:39:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 20 00:44:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 20 00:45:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2372c17b-d624-ee05-f8b6-bdc541467414 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880585ab2800, cur 1537429536 expire 1537429386 last 1537429309 Sep 20 00:50:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 20 00:51:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b3dbf956-7e89-0c1e-78ea-731a8bc6d52f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800280d6400, cur 1537429884 expire 1537429734 last 1537429657 Sep 20 00:55:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5b2d2223-3230-5ad8-6efb-f034c5d2024a (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bafc0000, cur 1537430153 expire 1537430003 last 1537429926 Sep 20 00:56:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Sep 20 00:57:42 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 00:57:42 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 20 01:07:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 20 01:07:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a64c8965-8cdf-3e26-ace0-67d99a3e9127 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807562d5800, cur 1537430859 expire 1537430709 last 1537430632 Sep 20 01:13:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aaf82144-888f-3910-3453-f1f28c997765 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a12a89c00, cur 1537431233 expire 1537431083 last 1537431006 Sep 20 01:14:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 20 01:21:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Sep 20 01:21:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client debf50a9-7fda-f5a5-d7fc-9d7e47664c4b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f5294cc00, cur 1537431686 expire 1537431536 last 1537431459 Sep 20 01:28:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client baeda5c5-6b81-fc0d-af5c-56c2f607d3e8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880017fb8000, cur 1537432094 expire 1537431944 last 1537431867 Sep 20 02:44:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a65b6a32-8838-2097-40bf-5f120478d909 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cde68f800, cur 1537436667 expire 1537436517 last 1537436440 Sep 20 02:45:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Sep 20 02:55:58 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 03:10:37 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 03:59:21 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 04:15:12 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 07:09:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2bf712ad-5c83-ae33-f321-ede358691a75 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880943c08800, cur 1537452544 expire 1537452394 last 1537452317 Sep 20 08:37:42 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 08:37:42 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 20 08:37:43 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 09:23:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a8e9ed8e-8f22-7b87-0349-f77f30896eb7 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe258ec00, cur 1537460611 expire 1537460461 last 1537460384 Sep 20 09:38:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4126d0f-01db-4510-bc7f-c4d6dc72b1c3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7c636800, cur 1537461515 expire 1537461365 last 1537461288 Sep 20 09:47:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 09:48:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 20 09:52:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b1304433-5329-2e61-fdcf-30f372712309 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096d3f7c00, cur 1537462338 expire 1537462188 last 1537462111 Sep 20 09:54:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 10:02:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0e8703a-7ea1-f606-c81d-440b264b960c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809c7569c00, cur 1537462953 expire 1537462803 last 1537462726 Sep 20 10:13:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 10:16:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 10:17:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e5c264f9-66b0-8a20-f7d6-021cb506ddd5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880353c3e400, cur 1537463820 expire 1537463670 last 1537463593 Sep 20 10:24:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2643065e-d250-a82c-fc05-f00204d96c67 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039bd93800, cur 1537464268 expire 1537464118 last 1537464041 Sep 20 10:38:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 10:49:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f45deea6-6f15-1006-e9ec-84bb4a133e54 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f82000, cur 1537465770 expire 1537465620 last 1537465543 Sep 20 10:50:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7748ded-adcc-8618-45ae-661e58335242 (at 10.8.28.4@o2ib6) in 165 seconds. I think it's dead, and I am evicting it. exp ffff88102fff7400, cur 1537465846 expire 1537465696 last 1537465681 Sep 20 10:58:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 11:04:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 11:04:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c937d90-cf76-1877-f2bf-c6771e65386d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f19cb4000, cur 1537466673 expire 1537466523 last 1537466446 Sep 20 11:10:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7748ded-adcc-8618-45ae-661e58335242 (at 10.8.28.4@o2ib6) Sep 20 11:19:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 20 11:19:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0e30549-cafa-5980-a6e0-b58560db51bb (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c93ff4c00, cur 1537467579 expire 1537467429 last 1537467352 Sep 20 11:32:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 69d08056-e251-2464-31b1-163f8fdeba3a (at 10.8.9.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882002f79c00, cur 1537468349 expire 1537468199 last 1537468122 Sep 20 11:40:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client abd70884-e925-a33f-5ede-8fb2bcd8014b (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880074ea9000, cur 1537468840 expire 1537468690 last 1537468613 Sep 20 11:40:43 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 11:40:43 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 20 11:47:48 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 11:47:48 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 20 11:47:49 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 11:47:49 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 20 11:48:23 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 11:49:44 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 11:49:44 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 20 11:52:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 69d08056-e251-2464-31b1-163f8fdeba3a (at 10.8.9.6@o2ib6) Sep 20 12:09:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fea67d76-2620-e1de-17d7-c8df7e587138 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85fcf800, cur 1537470570 expire 1537470420 last 1537470343 Sep 20 12:09:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 20 12:25:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ddba2ce-4876-a809-2b39-3f938b2c91b0 (at 10.8.9.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880074ea9c00, cur 1537471523 expire 1537471373 last 1537471296 Sep 20 12:30:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 13cdadc3-03d7-4d10-7d89-9f1c7f330d6f (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804c87d7c00, cur 1537471817 expire 1537471667 last 1537471590 Sep 20 12:30:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 12:32:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1e3c3c8f-7a90-a373-116e-abe5375db995 (at 10.9.114.11@o2ib4) Sep 20 12:32:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0291bf49-3c40-76aa-f27e-696f0f6f2a76 (at 10.8.28.11@o2ib6) Sep 20 12:33:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 77d246f8-967f-feff-4721-113faf2c7037 (at 10.9.114.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804541f0000, cur 1537471988 expire 1537471838 last 1537471761 Sep 20 12:33:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Sep 20 12:36:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e36707c-f955-e227-4ff1-fad8cf1a4b6f (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88130aae6c00, cur 1537472178 expire 1537472028 last 1537471951 Sep 20 12:36:18 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Sep 20 12:36:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 813af67a-4585-c6d9-9c57-ed2fda7e160a (at 10.9.112.6@o2ib4) Sep 20 12:39:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df909f04-6ce6-36ec-3e29-fbb52b19c896 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3ac400, cur 1537472374 expire 1537472224 last 1537472147 Sep 20 12:39:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 20 12:39:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 20 12:42:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 597b1b51-5ae5-7303-6ef6-47a17811a8c4 (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a45b74400, cur 1537472571 expire 1537472421 last 1537472344 Sep 20 12:42:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 12:44:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac7cce33-30e2-fda6-4577-5d0f043bc3ce (at 10.9.101.60@o2ib4) in 184 seconds. I think it's dead, and I am evicting it. exp ffff88200bed6400, cur 1537472647 expire 1537472497 last 1537472463 Sep 20 12:45:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca4de82a-a28c-8cdd-44d5-f14392933650 (at 10.8.9.1@o2ib6) Sep 20 12:46:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a093d0e4-d87a-3796-30df-b76ed0eb923a (at 10.8.28.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a3bec1c00, cur 1537472813 expire 1537472663 last 1537472586 Sep 20 12:48:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3421e93-fae8-00ca-fb76-98b8844f1714 (at 10.9.112.12@o2ib4) Sep 20 12:48:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 12:49:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10ef3673-afdb-3e3c-0639-a686fe26d7ea (at 10.9.113.5@o2ib4) Sep 20 12:49:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 20 12:49:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 54a034ac-890b-dfdb-eb9a-0244c2ffbf13 (at 10.8.14.3@o2ib6) reconnecting Sep 20 12:52:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e58be353-6bc0-1c69-b54d-1db7c01f7d82 (at 10.9.112.17@o2ib4) Sep 20 12:52:37 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 20 12:56:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3e98c37-6c62-f76f-b4c5-de4c69d99884 (at 10.9.112.14@o2ib4) Sep 20 12:56:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 20 12:59:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1281292c-097c-6e3d-602e-b2894d3cc394 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d9efacc00, cur 1537473555 expire 1537473405 last 1537473328 Sep 20 13:03:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7edfc4e1-499e-8b03-a014-f9e032f4d537 (at 10.8.28.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b7eafcc00, cur 1537473825 expire 1537473675 last 1537473598 Sep 20 13:05:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1fe4c082-b163-b490-c6c0-5cd56ff75ef7 (at 10.8.28.2@o2ib6) Sep 20 13:05:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 20 13:21:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fefcebc-3ba3-cd46-c617-8fd168dc2f9d (at 10.8.28.1@o2ib6) Sep 20 13:21:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 13:22:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 54a034ac-890b-dfdb-eb9a-0244c2ffbf13 (at 10.8.14.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88073018c000, cur 1537474928 expire 1537474778 last 1537474701 Sep 20 13:31:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 20 13:39:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ccf2cd2-3334-5ed7-fd16-8f419abb625e (at 10.9.113.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fa0443000, cur 1537475975 expire 1537475825 last 1537475748 Sep 20 13:39:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 13:56:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 208dc229-120b-fb25-969a-4251ce07291e (at 10.9.113.2@o2ib4) Sep 20 14:06:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8aeaf2c5-d60e-e60b-979e-772ed04970e5 (at 10.9.113.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c58c400, cur 1537477617 expire 1537477467 last 1537477390 Sep 20 14:15:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c5c172f6-7977-6a5d-87c0-d8ce5d77408c (at 10.9.114.1@o2ib4) Sep 20 14:16:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a8fb93a6-0416-ae43-f2a3-02ae33b50dff (at 10.9.114.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88189aa8f800, cur 1537478187 expire 1537478037 last 1537477960 Sep 20 14:22:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9c703285-c8c2-4d01-1888-2b09c1187ec2 (at 10.9.113.7@o2ib4) Sep 20 14:24:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9735ccc7-8259-ff1c-fd85-909e0c4749e6 (at 10.9.114.14@o2ib4) Sep 20 14:24:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94a51969-8bf2-aecc-0815-c33c5fc6774a (at 10.9.114.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881860650c00, cur 1537478694 expire 1537478544 last 1537478467 Sep 20 14:24:54 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Sep 20 14:28:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9730dd45-13b7-8e7b-8a96-57b1c2946c2e (at 10.9.114.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e750cd800, cur 1537478938 expire 1537478788 last 1537478711 Sep 20 14:34:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acb78d5c-ce52-9a44-b71b-f19d6c82b7dc (at 10.8.16.6@o2ib6) Sep 20 14:36:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1e3c3c8f-7a90-a373-116e-abe5375db995 (at 10.9.114.11@o2ib4) Sep 20 14:36:23 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 20 14:41:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f8e054ac-159d-5b1f-e0ab-33ac7d88fa34 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2d630000, cur 1537479710 expire 1537479560 last 1537479483 Sep 20 14:48:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7d4433d8-921c-0a7c-3ccb-26ba1ef18d63 (at 10.8.9.4@o2ib6) Sep 20 14:48:50 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 20 14:49:06 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 14:56:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9ee4251f-89f3-8c74-c2bb-ce40826a23fa (at 10.9.112.9@o2ib4) Sep 20 14:56:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 14:56:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 22449e06-119f-e4cb-f0dd-78a0ef34b03b (at 10.9.112.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc4b400, cur 1537480599 expire 1537480449 last 1537480372 Sep 20 14:57:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ac59e9e-91ba-4e58-e346-2984e55e2f86 (at 10.8.29.3@o2ib6) in 156 seconds. I think it's dead, and I am evicting it. exp ffff880f93131000, cur 1537480675 expire 1537480525 last 1537480519 Sep 20 15:15:22 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 20 15:21:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7ef9ab91-de0a-6e83-ff2e-c82de2e8e0a8 (at 10.8.13.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e1c00, cur 1537482063 expire 1537481913 last 1537481836 Sep 20 15:39:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4966203-761f-6dc5-9911-bf9e91bbb908 (at 10.8.13.24@o2ib6) Sep 20 15:39:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 15:51:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 15:51:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0561bfe9-fbff-3aff-8e95-dcf37a6d3212 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880343185000, cur 1537483910 expire 1537483760 last 1537483683 Sep 20 15:54:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef8a3c6e-b8a7-865d-f1d0-ae5824792f80 (at 10.9.115.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001d9e000, cur 1537484073 expire 1537483923 last 1537483846 Sep 20 16:10:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6f11ebc-0bd6-ad8b-390a-e041d9836de8 (at 10.9.115.6@o2ib4) Sep 20 16:30:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d2af66e-b8bc-c7ae-31fe-f0073ae62f1a (at 10.9.112.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187f2b5c00, cur 1537486218 expire 1537486068 last 1537485991 Sep 20 16:34:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d72d812-c7d5-7994-46e8-37cc30b02c3b (at 10.9.113.10@o2ib4) Sep 20 16:34:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b815a4ee-e232-f255-a078-26f6c7d11491 (at 10.9.113.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881999302800, cur 1537486460 expire 1537486310 last 1537486233 Sep 20 16:42:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ad0805c-695a-06f8-134f-80e986613e97 (at 10.8.29.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004a19b000, cur 1537486948 expire 1537486798 last 1537486721 Sep 20 16:43:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 229af665-eaec-49e5-b8c2-2c97d6390c53 (at 10.9.112.8@o2ib4) Sep 20 16:43:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f74b4044-1fa9-b3c4-ea3c-bc64516ad55d (at 10.9.112.8@o2ib4) in 226 seconds. I think it's dead, and I am evicting it. exp ffff88130ffd5c00, cur 1537487024 expire 1537486874 last 1537486798 Sep 20 16:46:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 23cf492c-730a-7bf7-a7e3-af1d811b9767 (at 10.9.112.11@o2ib4) Sep 20 17:00:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cdadf045-35ed-25fc-1b8b-f68ab75f9e02 (at 10.8.29.4@o2ib6) Sep 20 17:21:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e3ce49d5-da8c-e9c3-95c1-7aa40d9f6b9a (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031dc70800, cur 1537489298 expire 1537489148 last 1537489071 Sep 20 17:27:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a254e27-51f4-d105-5121-4ead09439d09 (at 10.8.29.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88070479ac00, cur 1537489648 expire 1537489498 last 1537489421 Sep 20 17:27:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 17:37:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Sep 20 17:38:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9425f16-111c-0989-b4a7-95612c24ae12 (at 10.8.29.1@o2ib6) Sep 20 17:40:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6af2ea1f-90a9-a9e4-b745-34959e7c03c1 (at 10.9.112.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881595252000, cur 1537490439 expire 1537490289 last 1537490212 Sep 20 17:40:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 229af665-eaec-49e5-b8c2-2c97d6390c53 (at 10.9.112.8@o2ib4) Sep 20 17:43:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 78f304a9-ef70-69c1-d13c-5eb1d7373efa (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd819800, cur 1537490635 expire 1537490485 last 1537490408 Sep 20 17:43:55 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 20 17:44:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6c260e25-84e3-f0df-7485-52b1ed7d2cd1 (at 10.8.29.5@o2ib6) Sep 20 17:44:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 17:48:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 20 17:52:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 84c65082-dbe2-f2f8-e843-8d09b8b4d4b3 (at 10.9.112.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b4463dc00, cur 1537491144 expire 1537490994 last 1537490917 Sep 20 17:57:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9735ccc7-8259-ff1c-fd85-909e0c4749e6 (at 10.9.114.14@o2ib4) Sep 20 17:58:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6df9f485-8ac4-4758-2045-ffad4e1b85f1 (at 10.8.29.3@o2ib6) Sep 20 17:58:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 20 17:59:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1d20ca5d-a243-1ba7-0f2c-c9bcc28c5432 (at 10.8.16.3@o2ib6) Sep 20 17:59:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 18:00:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f0cad8f-39e8-28c5-6a91-001c1bc26264 (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8ca9dc00, cur 1537491636 expire 1537491486 last 1537491409 Sep 20 18:00:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 18:01:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b4d5361-ba1f-4115-4c98-fa9231cae0c7 (at 10.8.15.6@o2ib6) Sep 20 18:04:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c7eefa3-fc99-540c-06e7-6b75b4a7596a (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200bed5800, cur 1537491848 expire 1537491698 last 1537491621 Sep 20 18:09:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fc8cc8b-dbac-7a5a-c932-7b029961ec9f (at 10.9.112.10@o2ib4) Sep 20 18:09:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 18:18:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8fe6cf2c-8fb9-7f07-9d12-2ee19c744074 (at 10.8.29.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93130800, cur 1537492733 expire 1537492583 last 1537492506 Sep 20 18:18:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 18:21:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9f0cad8f-39e8-28c5-6a91-001c1bc26264 (at 10.9.114.5@o2ib4) Sep 20 18:21:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 18:36:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bb612a21-8c04-ff05-d4d6-29a9ab6ae3a3 (at 10.8.29.2@o2ib6) Sep 20 18:49:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a10887d8-b417-4aea-2a92-dc77a22f0e77 (at 10.8.9.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a9368400, cur 1537494559 expire 1537494409 last 1537494332 Sep 20 18:57:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 64986085-e692-3fd6-20df-eeb9b52bb1ae (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103b5f0c00, cur 1537495067 expire 1537494917 last 1537494840 Sep 20 19:01:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28545079-4186-3c7e-b563-229081acfbe4 (at 10.8.19.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe2588800, cur 1537495269 expire 1537495119 last 1537495042 Sep 20 19:06:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 06717f7d-e3d0-b41d-98d4-62cae162f633 (at 10.9.113.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808cf36ec00, cur 1537495597 expire 1537495447 last 1537495370 Sep 20 19:11:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09224f28-d4e9-41eb-4826-893251222ec0 (at 10.8.9.10@o2ib6) Sep 20 19:15:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 046b6f7b-5531-8a41-d540-00485b286155 (at 10.8.15.3@o2ib6) Sep 20 19:17:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb5d83f7-760e-543e-9b43-69605647f829 (at 10.8.19.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f97759c00, cur 1537496222 expire 1537496072 last 1537495995 Sep 20 19:22:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b94da0a0-014c-9109-1d28-9e062cace35c (at 10.9.113.3@o2ib4) Sep 20 19:25:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 735ed23a-b16b-155e-0bf7-7d4e1e35f2a6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eebfacc00, cur 1537496711 expire 1537496561 last 1537496484 Sep 20 19:32:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ddff929a-cc60-fb27-9b8b-1a40efc890e0 (at 10.8.19.3@o2ib6) Sep 20 19:32:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 19:33:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 705dedf5-6e74-2df9-26cb-219d56bbecda (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e20c76400, cur 1537497202 expire 1537497052 last 1537496975 Sep 20 19:43:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 19:44:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 381a5aa8-987e-3841-1a8a-d635d3fa6ed1 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880526a79c00, cur 1537497873 expire 1537497723 last 1537497646 Sep 20 19:47:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd00204e-0369-8a5e-ce93-11c40307f470 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88071e052400, cur 1537498079 expire 1537497929 last 1537497852 Sep 20 19:51:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fb2820d7-aa84-d728-255b-3ad812e9e313 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040e0d400, cur 1537498277 expire 1537498127 last 1537498050 Sep 20 20:03:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3f233d6-3f0f-1910-c6b1-c29fe6231a1f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e2f35f400, cur 1537499009 expire 1537498859 last 1537498782 Sep 20 20:06:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f290ca8-b6f3-4578-79dc-5ce2e2ced910 (at 10.8.29.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099725a800, cur 1537499210 expire 1537499060 last 1537498983 Sep 20 20:06:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 20:18:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 20:18:13 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 20 20:21:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fd8236e-050b-c62d-2664-f6711708a9ca (at 10.8.29.6@o2ib6) Sep 20 20:22:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a0d9a95f-6ae7-9746-7a80-aae2dae85b8e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d442ea400, cur 1537500154 expire 1537500004 last 1537499927 Sep 20 20:22:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 20:26:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 20:26:16 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 20 20:26:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2075656-9dc9-d0d2-5596-9a2c1a72e1af (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88055c14e800, cur 1537500390 expire 1537500240 last 1537500163 Sep 20 20:31:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 840370bd-099f-dcd6-646d-47fcf77bc5e6 (at 10.8.19.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e25f6d000, cur 1537500670 expire 1537500520 last 1537500443 Sep 20 20:32:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7f7971f-76aa-1ebc-519c-3c7d24cd7284 (at 10.8.17.26@o2ib6) in 195 seconds. I think it's dead, and I am evicting it. exp ffff880461425c00, cur 1537500746 expire 1537500596 last 1537500551 Sep 20 20:34:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 430f70a6-735b-dec3-13cf-4f919b0b9772 (at 10.8.9.9@o2ib6) Sep 20 21:07:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 21:09:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b73ef9e9-7e22-3ca6-6c8b-bcb0a26cebc3 (at 10.9.112.4@o2ib4) Sep 20 21:11:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b73ef9e9-7e22-3ca6-6c8b-bcb0a26cebc3 (at 10.9.112.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880970fbbc00, cur 1537503063 expire 1537502913 last 1537502836 Sep 20 21:11:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 21:28:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 21:32:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51f7856d-d4fc-3e48-c3ad-ddf4a2ff71e5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc3ef9c00, cur 1537504362 expire 1537504212 last 1537504135 Sep 20 21:32:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 20 21:47:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 21:50:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e04fbaae-c141-2ddd-e08f-482ea07cb44e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806d1953c00, cur 1537505456 expire 1537505306 last 1537505229 Sep 20 21:54:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 21:58:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7a7a4a9-1412-40a9-f4e7-d10253e48ce1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d288bcc00, cur 1537505918 expire 1537505768 last 1537505691 Sep 20 21:59:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 22:03:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3ecc99a-c3d0-2c69-1b7c-c6ee2bfc28b2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fbafe6c00, cur 1537506182 expire 1537506032 last 1537505955 Sep 20 22:14:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 22:17:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 22:18:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1bea12ef-d836-7bef-ec02-15979daf7540 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68f800, cur 1537507091 expire 1537506941 last 1537506864 Sep 20 22:21:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 20 22:21:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 898f0c39-920d-9799-b89f-3722981ebcb2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a16a80400, cur 1537507316 expire 1537507166 last 1537507089 Sep 20 22:23:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74018050-881d-5ba1-4999-4b91e5dc4c47 (at 10.8.20.19@o2ib6) in 210 seconds. I think it's dead, and I am evicting it. exp ffff880ea36f8c00, cur 1537507392 expire 1537507242 last 1537507182 Sep 20 22:24:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 009a5011-ebf1-e2df-60bc-a300020f92e6 (at 10.8.17.26@o2ib6) in 175 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e5e400, cur 1537507468 expire 1537507318 last 1537507293 Sep 20 22:25:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 22:31:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 22:32:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ed20c0d-7011-8270-b818-5758be8528a3 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d169c8000, cur 1537507960 expire 1537507810 last 1537507733 Sep 20 22:38:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 22:38:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a0b9683-efef-df82-e217-ac3326cc2912 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804f43d2400, cur 1537508338 expire 1537508188 last 1537508111 Sep 20 22:45:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 22:46:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c2b5b8c5-f60a-39b0-31a8-e60ef5ee58b4 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6d1df800, cur 1537508765 expire 1537508615 last 1537508538 Sep 20 22:53:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 22:54:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db0a585f-fa0f-6ecf-189c-cd660a8bfbef (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272376800, cur 1537509283 expire 1537509133 last 1537509056 Sep 20 23:01:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b008030-a579-ea7a-e225-c8683eb52ccf (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800190c1800, cur 1537509663 expire 1537509513 last 1537509436 Sep 20 23:09:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 23:16:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 23:17:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ee431512-4ab9-089c-85e8-e3b49e34f9fb (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6d1ddc00, cur 1537510672 expire 1537510522 last 1537510445 Sep 20 23:26:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d84fe2cf-7a17-58ac-cc3e-ee2b25b92edc (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edfe000, cur 1537511218 expire 1537511068 last 1537510991 Sep 20 23:35:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 23:46:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 20 23:47:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 73b62899-72ff-7fd1-1a10-bb9b8e067883 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b28fd9c00, cur 1537512449 expire 1537512299 last 1537512222 Sep 20 23:57:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c0bb6c9-6e34-bef0-1199-48afe0f116ab (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d5cb800, cur 1537513036 expire 1537512886 last 1537512809 Sep 20 23:57:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 00:01:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 00:01:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 999f9ddc-61ce-e5a9-0d94-3e4efa52c373 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040d10400, cur 1537513292 expire 1537513142 last 1537513065 Sep 21 00:08:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 00:09:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3fd85116-06fe-b91a-8de6-722f50318d14 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809409c5000, cur 1537513750 expire 1537513600 last 1537513523 Sep 21 00:18:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2aa0d95-1594-b5b6-876b-0b4c0ab4ab26 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88075628a800, cur 1537514296 expire 1537514146 last 1537514069 Sep 21 00:24:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5c7e688-753c-1697-4a13-2d1c35b17a75 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88021630f800, cur 1537514690 expire 1537514540 last 1537514463 Sep 21 00:27:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 00:31:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 947c3fe5-68a9-2330-3884-8e6239384242 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800246f9c00, cur 1537515091 expire 1537514941 last 1537514864 Sep 21 00:33:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 00:37:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 19faf2ca-aa3e-0a57-ccf1-0709f0a53de9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fb446800, cur 1537515451 expire 1537515301 last 1537515224 Sep 21 00:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 00:40:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 00:42:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7d4433d8-921c-0a7c-3ccb-26ba1ef18d63 (at 10.8.9.4@o2ib6) Sep 21 00:47:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ed19ed0d-f982-b5bb-8f50-bdd5ebd10604 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803d1771800, cur 1537516038 expire 1537515888 last 1537515811 Sep 21 00:47:18 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 00:55:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:00:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05f02435-2b82-b12d-f60a-ba29e119e04c (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88022a075000, cur 1537516804 expire 1537516654 last 1537516577 Sep 21 01:09:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:12:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7abff122-dae2-66d5-58dc-feeb2ee8a24a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b275a9c00, cur 1537517603 expire 1537517453 last 1537517376 Sep 21 01:30:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:34:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:35:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2de5d149-60a9-19ff-8770-169edb70415d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c1514800, cur 1537518912 expire 1537518762 last 1537518685 Sep 21 01:35:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 01:39:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cc366165-1e1d-2c17-2386-fe49a24d8fdb (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe9df6800, cur 1537519145 expire 1537518995 last 1537518918 Sep 21 01:47:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:55:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:56:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f50e9ef9-66b2-e921-67d2-5dd0b78faff5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5cf02000, cur 1537520173 expire 1537520023 last 1537519946 Sep 21 01:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 01:59:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dec14a19-dd13-d0b0-9bbd-be03441f6faf (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102cfd9800, cur 1537520377 expire 1537520227 last 1537520150 Sep 21 02:03:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 11d11c2f-75cf-7862-20cb-af3e74416bde (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b5f37d400, cur 1537520639 expire 1537520489 last 1537520412 Sep 21 02:12:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 02:22:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f526d8a-647d-a5c4-310e-a9f90deb5e50 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ba78c3800, cur 1537521754 expire 1537521604 last 1537521527 Sep 21 02:27:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 02:59:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac5b86b0-4638-55b1-86fe-844bb4ffa468 (at 10.8.27.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880761fcac00, cur 1537523966 expire 1537523816 last 1537523739 Sep 21 03:14:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 03:15:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 148fd41e-43ed-5be5-441c-8d48a9042e0c (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88057a352c00, cur 1537524907 expire 1537524757 last 1537524680 Sep 21 03:19:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8533e8b-3a11-3aef-6b11-f9c53bf9ac57 (at 10.8.27.21@o2ib6) Sep 21 03:42:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a8c48468-b6e5-6403-9a8d-4c7ccdb0aa8d (at 10.9.113.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c589000, cur 1537526569 expire 1537526419 last 1537526342 Sep 21 03:47:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99578155-d31b-f5fb-cdca-74b401e806a0 (at 10.12.4.70@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810302f3c00, cur 1537526873 expire 1537526723 last 1537526646 Sep 21 04:00:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 199b5230-b1f8-9a0e-4dab-d24f3b9586f8 (at 10.8.26.16@o2ib6) reconnecting Sep 21 04:00:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 199b5230-b1f8-9a0e-4dab-d24f3b9586f8 (at 10.8.26.16@o2ib6) Sep 21 04:03:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a8c48468-b6e5-6403-9a8d-4c7ccdb0aa8d (at 10.9.113.13@o2ib4) Sep 21 04:22:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 04:28:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5d056df2-96d6-ae71-7305-f05930868eb5 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881463fe6c00, cur 1537529339 expire 1537529189 last 1537529112 Sep 21 04:46:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Sep 21 04:46:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a46c467a-9f53-b135-92c6-44c55506a9ad (at 10.9.113.1@o2ib4) Sep 21 04:46:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2f1b044-d7c9-a64d-1a65-913ccb6ed1b6 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9f800, cur 1537530412 expire 1537530262 last 1537530185 Sep 21 04:46:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 04:54:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 04:54:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49a9d726-2234-14e8-18fe-ac1ceb4fde76 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003f6cf000, cur 1537530897 expire 1537530747 last 1537530670 Sep 21 04:57:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 04:58:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 82b231a5-8183-5735-84fd-9f06561e1714 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800670c7400, cur 1537531101 expire 1537530951 last 1537530874 Sep 21 05:02:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1616839a-7566-2401-6d6b-559f1c69e5a3 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880017fbc400, cur 1537531379 expire 1537531229 last 1537531152 Sep 21 05:03:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Sep 21 05:07:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 21 05:53:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 05:54:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ad2e736-10df-4a07-d3a6-2e03fd295c0a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c62fa000, cur 1537534458 expire 1537534308 last 1537534231 Sep 21 05:56:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 05:57:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 126246d9-6294-3079-7dea-27b8c9c4fc72 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809439dfc00, cur 1537534660 expire 1537534510 last 1537534433 Sep 21 06:02:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 06:03:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28acc3fe-1844-99cd-e1da-e5acf850d07a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880456371400, cur 1537535014 expire 1537534864 last 1537534787 Sep 21 06:06:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 06:07:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5c24d820-e587-39a3-d638-4a712f9185ab (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806be21d400, cur 1537535229 expire 1537535079 last 1537535002 Sep 21 06:10:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b243632-8e2a-ce00-84ec-87f794c3e7f4 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802955b0000, cur 1537535421 expire 1537535271 last 1537535194 Sep 21 06:18:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 06:19:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to db0130ba-6755-04c2-1981-5eeca0fbcd17 (at 10.9.113.15@o2ib4) Sep 21 06:20:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3a71646-95ca-347c-eb7c-fae83a2cd967 (at 10.9.113.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e0000, cur 1537536014 expire 1537535864 last 1537535787 Sep 21 06:29:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65aee6b5-1098-7f78-f07e-6954602bce37 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c0ea43000, cur 1537536588 expire 1537536438 last 1537536361 Sep 21 06:31:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0c061e1-3359-0b2c-9967-235f54fc7dbc (at 10.8.14.1@o2ib6) in 166 seconds. I think it's dead, and I am evicting it. exp ffff8801a936b400, cur 1537536664 expire 1537536514 last 1537536498 Sep 21 06:38:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 06:52:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db60cf87-3eb2-4fb2-6261-299a733eb78a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880761fcac00, cur 1537537921 expire 1537537771 last 1537537694 Sep 21 07:00:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 07:06:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a0edf415-e4ce-ea02-5d81-71aa79cbc8cb (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092a98e000, cur 1537538812 expire 1537538662 last 1537538585 Sep 21 07:10:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b900f5d9-429f-ad3b-0530-6a4bcbfd6266 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eaa678000, cur 1537539026 expire 1537538876 last 1537538799 Sep 21 07:19:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 07:24:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 07:24:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ae3656a-d8b9-8053-198b-51e8f0257d3e (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c562b3400, cur 1537539898 expire 1537539748 last 1537539671 Sep 21 07:27:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 21 07:34:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 35f59bc7-6242-c18e-13cb-2e3d9881655a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802ca27a400, cur 1537540454 expire 1537540304 last 1537540227 Sep 21 07:42:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 07:53:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 07:54:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bac66f5e-c2db-4477-e56a-d7ab5f0ab6d5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eaa67e400, cur 1537541669 expire 1537541519 last 1537541442 Sep 21 08:01:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 08:02:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 794f4bdf-33b5-e1f2-d99c-c40e856d7b24 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4b597000, cur 1537542142 expire 1537541992 last 1537541915 Sep 21 08:07:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 08:07:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 164d13b4-6acb-9586-550f-96d0ecbbcad6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6c8d9800, cur 1537542470 expire 1537542320 last 1537542243 Sep 21 08:17:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 08:21:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 08:22:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3d3f4ff9-e5de-838b-7379-4c7852643f99 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053213800, cur 1537543322 expire 1537543172 last 1537543095 Sep 21 08:22:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 08:24:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 08:25:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d10eaed5-3c08-23de-b4c6-810f3bf5ad80 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dc4522c00, cur 1537543526 expire 1537543376 last 1537543299 Sep 21 09:15:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e3676b2-4303-31c8-f22c-9a2bd4038401 (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880071c0cc00, cur 1537546559 expire 1537546409 last 1537546332 Sep 21 09:17:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 229af665-eaec-49e5-b8c2-2c97d6390c53 (at 10.9.112.8@o2ib4) Sep 21 09:29:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client beaa3745-703f-7a67-700f-ca8c2451fd8f (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881849b38800, cur 1537547399 expire 1537547249 last 1537547172 Sep 21 09:29:59 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Sep 21 09:30:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbb9db85-60f5-adbd-5b0f-c1781850eda7 (at 10.8.19.8@o2ib6) Sep 21 09:31:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to db0130ba-6755-04c2-1981-5eeca0fbcd17 (at 10.9.113.15@o2ib4) Sep 21 09:32:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b8b40ce8-0e35-0947-d1cb-c4e2b1387484 (at 10.9.115.4@o2ib4) Sep 21 09:32:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Sep 21 09:33:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9ee4251f-89f3-8c74-c2bb-ce40826a23fa (at 10.9.112.9@o2ib4) Sep 21 09:35:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 430f70a6-735b-dec3-13cf-4f919b0b9772 (at 10.8.9.9@o2ib6) Sep 21 09:36:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 511cf0f1-3237-5998-59bf-81a32d3d5f9c (at 10.9.114.4@o2ib4) Sep 21 09:37:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30707d31-ee5f-ca72-3ac0-5f805897b092 (at 10.9.114.3@o2ib4) Sep 21 09:43:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 954c7088-e35d-99d2-4b0b-d1fb12dafd98 (at 10.9.113.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882002f7b400, cur 1537548200 expire 1537548050 last 1537547973 Sep 21 09:52:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fe0ee6f3-c9d3-a009-f66c-29f578585e2d (at 10.8.6.12@o2ib6) Sep 21 09:52:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 09:53:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Sep 21 10:02:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 954c7088-e35d-99d2-4b0b-d1fb12dafd98 (at 10.9.113.14@o2ib4) Sep 21 10:23:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 401672aa-cae4-0521-441a-2ae010fbf8bb (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880071c0c000, cur 1537550630 expire 1537550480 last 1537550403 Sep 21 10:42:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cfb68c30-ab78-dd4f-da36-e9ec9e167d7e (at 10.9.112.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca48800, cur 1537551739 expire 1537551589 last 1537551512 Sep 21 11:03:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cfb68c30-ab78-dd4f-da36-e9ec9e167d7e (at 10.9.112.5@o2ib4) Sep 21 11:06:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55873861-e524-0a33-b22c-845119aa1ad3 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881868a08400, cur 1537553166 expire 1537553016 last 1537552939 Sep 21 11:21:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c40808fa-a89a-ff5b-2c9d-153f7d1389a6 (at 10.9.113.6@o2ib4) Sep 21 11:26:30 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 11:50:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4c2f9201-3d96-2049-8bb6-0d530a17263b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015ca3c00, cur 1537555835 expire 1537555685 last 1537555608 Sep 21 11:54:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 11:58:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75b83641-cdab-8e37-f427-659d3e9db4e0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bc635a400, cur 1537556295 expire 1537556145 last 1537556068 Sep 21 11:59:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 12:00:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8dbd6139-2097-ab63-d9a3-e6c5d4e12db0 (at 10.9.114.6@o2ib4) Sep 21 12:02:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f63fd811-3817-1413-d06f-d772e01e3d81 (at 10.9.114.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881800346000, cur 1537556524 expire 1537556374 last 1537556297 Sep 21 12:03:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f8d03caa-ed3e-1287-ca04-4370dc0743a0 (at 10.210.47.253@o2ib3) in 154 seconds. I think it's dead, and I am evicting it. exp ffff88102dfe9800, cur 1537556600 expire 1537556450 last 1537556446 Sep 21 12:07:11 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 10 seconds Sep 21 12:07:11 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 3 previous similar messages Sep 21 12:17:05 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 4 seconds Sep 21 12:22:08 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 7 seconds Sep 21 12:22:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d9ffbd7-74a0-eda1-9dc4-702facd5ce9b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d288b8c00, cur 1537557742 expire 1537557592 last 1537557515 Sep 21 12:25:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 12:27:11 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 10 seconds Sep 21 12:29:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 161c26cd-6d4c-1947-0f0c-1c430651be4f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880010e9f000, cur 1537558161 expire 1537558011 last 1537557934 Sep 21 12:32:01 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 0 seconds Sep 21 12:37:04 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 3 seconds Sep 21 12:41:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 12:42:07 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 6 seconds Sep 21 12:46:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 21a40b00-b522-bb4b-5f79-3ee6d35dbfad (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f196e0400, cur 1537559187 expire 1537559037 last 1537558960 Sep 21 12:50:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 12:50:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 06f1a0c3-2a5d-81ff-4991-81d931c28c01 (at 10.8.1.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb4a7c400, cur 1537559412 expire 1537559262 last 1537559185 Sep 21 12:51:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99578155-d31b-f5fb-cdca-74b401e806a0 (at 10.12.4.70@o2ib) Sep 21 13:01:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:01:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 13:02:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1fc76a4e-fa36-d8fa-760b-2503486f5589 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f34ac5000, cur 1537560132 expire 1537559982 last 1537559905 Sep 21 13:07:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e72610c3-9d54-955d-d1b7-6c2d7a3698b6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88015bc8d400, cur 1537560447 expire 1537560297 last 1537560220 Sep 21 13:08:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:13:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:14:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 82bc00b5-3223-8c13-4ab9-07dada141843 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e6bf800, cur 1537560856 expire 1537560706 last 1537560629 Sep 21 13:19:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d9f8496-25eb-94d2-eee6-1036dac16b44 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1a800, cur 1537561159 expire 1537561009 last 1537560932 Sep 21 13:26:00 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 13:26:01 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 13:26:01 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 41 previous similar messages Sep 21 13:26:03 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 13:26:03 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 152 previous similar messages Sep 21 13:26:05 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 13:26:05 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 278 previous similar messages Sep 21 13:27:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:33:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a8a9c3a-cafd-91f9-1e4d-42c1322e9ed3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9e800, cur 1537562009 expire 1537561859 last 1537561782 Sep 21 13:42:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:47:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:47:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 07524945-9a65-ffd3-0de4-e41430a48763 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880073b30000, cur 1537562879 expire 1537562729 last 1537562652 Sep 21 13:50:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:50:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 544216ea-8a2d-cc6a-14f5-c0fb1a8909ed (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88020cea6800, cur 1537563058 expire 1537562908 last 1537562831 Sep 21 13:53:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:54:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0d52f037-d057-7c46-c347-a58e74bd6db9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880526a7a800, cur 1537563241 expire 1537563091 last 1537563014 Sep 21 13:58:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 13:58:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75127976-3934-dda6-fa36-c13fd0c9ca1e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4a35d800, cur 1537563531 expire 1537563381 last 1537563304 Sep 21 14:01:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 14:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d8d517f2-e9c2-e6e3-66f9-39efeaba3a2a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c27d5800, cur 1537563720 expire 1537563570 last 1537563493 Sep 21 14:05:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 14:06:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 01b76e31-edf7-1829-685f-f479d932baff (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880afabaf800, cur 1537564002 expire 1537563852 last 1537563775 Sep 21 14:12:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 14:12:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2abe3c4-1a45-35e1-8573-65c2500add80 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc9771800, cur 1537564348 expire 1537564198 last 1537564121 Sep 21 14:17:14 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 12 seconds Sep 21 14:18:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aee13b7a-2407-96e5-7d57-fbe9ddd9256e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058c461800, cur 1537564738 expire 1537564588 last 1537564511 Sep 21 14:22:04 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 2 seconds Sep 21 14:26:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 14:27:07 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 5 seconds Sep 21 14:30:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 14:31:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 666f030c-e896-222a-2eda-b9b7c7cac369 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800500c1c00, cur 1537565495 expire 1537565345 last 1537565268 Sep 21 14:31:35 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 21 14:32:10 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 8 seconds Sep 21 14:34:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1d466413-418e-db82-5337-6a2f078692ec (at 10.8.19.2@o2ib6) Sep 21 14:34:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ac7b8475-251b-16ac-f682-d7b7d6e192fd (at 10.8.19.6@o2ib6) Sep 21 14:34:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 21 14:36:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 14:37:13 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 11 seconds Sep 21 14:37:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client abc3bcc1-4d3b-31cf-3564-0f0172b4e0d8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a98401400, cur 1537565835 expire 1537565685 last 1537565608 Sep 21 14:38:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9effb4e8-f195-a3eb-e68a-859b1f5dda95 (at 10.8.14.4@o2ib6) Sep 21 14:38:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 14:42:03 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.201@o2ib5: 1 seconds Sep 21 14:43:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 751dfad8-0583-f62d-7589-36a5d0421ab1 (at 10.9.112.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7797800, cur 1537566198 expire 1537566048 last 1537565971 Sep 21 14:43:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 14:43:42 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 21 14:44:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6ee21cad-8516-9d45-b8b5-14bf8c9828eb (at 10.8.20.19@o2ib6) in 214 seconds. I think it's dead, and I am evicting it. exp ffff880adae72c00, cur 1537566274 expire 1537566124 last 1537566060 Sep 21 15:00:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 229af665-eaec-49e5-b8c2-2c97d6390c53 (at 10.9.112.8@o2ib4) Sep 21 15:00:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 463f9e44-28eb-068b-2bc4-363a41fe3da4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806b59f4800, cur 1537567252 expire 1537567102 last 1537567025 Sep 21 15:05:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d432b743-edce-9169-cba9-6a12548dd0fb (at 10.9.101.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e71cd000, cur 1537567541 expire 1537567391 last 1537567314 Sep 21 15:10:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bcb3952c-15e5-559d-e361-b538eacdc535 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800496b0400, cur 1537567812 expire 1537567662 last 1537567585 Sep 21 15:12:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 15:12:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 15:17:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e1b86e1-19da-67a0-0adb-ff20ea98ec4f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5fcf2c00, cur 1537568236 expire 1537568086 last 1537568009 Sep 21 15:21:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 15a5638a-15d8-856e-6d91-d09b54ee6302 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5bc58c00, cur 1537568495 expire 1537568345 last 1537568268 Sep 21 15:25:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 15:25:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 21 15:26:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d5cd8b91-c8ee-31fa-74d7-16d8adbd2d7c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88027dad1000, cur 1537568776 expire 1537568626 last 1537568549 Sep 21 15:31:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 766ea08f-201c-3ca4-e701-314ffd221e46 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880562b49000, cur 1537569093 expire 1537568943 last 1537568866 Sep 21 15:36:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 15:36:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 15:46:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bafa50b2-1a16-2a88-1bd6-daff54476939 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88955400, cur 1537569969 expire 1537569819 last 1537569742 Sep 21 15:46:09 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 21 15:49:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 21 15:49:54 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 21 15:57:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 21f2bdb4-e599-9a2e-c402-6478230091f7 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582304400, cur 1537570621 expire 1537570471 last 1537570394 Sep 21 15:57:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 16:24:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1c18b613-f337-cfa9-1c98-fcc5edc3d913 (at 10.8.27.35@o2ib6) Sep 21 16:24:39 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 21 16:34:02 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 16:34:02 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 334 previous similar messages Sep 21 17:00:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac7dd2e6-957c-7138-459f-342637e2dd2b (at 10.12.4.58@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fe24c00, cur 1537574438 expire 1537574288 last 1537574211 Sep 21 17:00:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 17:08:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6cfe27ae-cac0-b498-3c9b-433a8afbbe02 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227cd000, cur 1537574930 expire 1537574780 last 1537574703 Sep 21 17:35:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 17:35:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 69be935b-b3a4-f1ee-4d16-df6e221b8061 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058c467000, cur 1537576549 expire 1537576399 last 1537576322 Sep 21 18:01:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 18:02:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7229a65-2a90-e97d-9965-3507a0653e4a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880756289800, cur 1537578159 expire 1537578009 last 1537577932 Sep 21 18:02:40 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 18:02:43 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 18:02:46 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 18:08:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 606840f7-5193-ec8f-4ebf-cd5a23c82488 (at 10.9.114.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b275ae800, cur 1537578520 expire 1537578370 last 1537578293 Sep 21 18:10:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 18:11:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db0efbdd-1311-0525-f056-3f47c92530b9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd64bc00, cur 1537578689 expire 1537578539 last 1537578462 Sep 21 18:11:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 18:23:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 872d2d0e-5312-c3a6-6c5e-c1460c457187 (at 10.8.19.4@o2ib6) Sep 21 18:24:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 18:25:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ba513a95-a411-2e35-8ee1-c76b78fd2188 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004aecd400, cur 1537579507 expire 1537579357 last 1537579280 Sep 21 18:25:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c17c7504-f891-a009-27ca-0f6544f6e1e4 (at 10.9.114.13@o2ib4) Sep 21 18:49:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 18:50:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 243ca9d9-e746-f3b9-b09f-330bb6bd49b0 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e810c00, cur 1537581021 expire 1537580871 last 1537580794 Sep 21 18:52:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 18:53:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6117ad5-47d6-75ce-2275-fa2e775aa6d9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea1800, cur 1537581239 expire 1537581089 last 1537581012 Sep 21 19:07:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:08:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c06ea8f2-1aad-c19a-66bf-f1da9bbf3315 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c46160400, cur 1537582095 expire 1537581945 last 1537581868 Sep 21 19:13:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:14:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9261e5f5-87dc-544e-9631-5b3d28333b34 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802267e0800, cur 1537582465 expire 1537582315 last 1537582238 Sep 21 19:25:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:25:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 750e09cc-46df-c61a-073a-27af1eef18c5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cb95ee800, cur 1537583159 expire 1537583009 last 1537582932 Sep 21 19:29:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:30:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client af39ecaa-504a-e495-abc9-ba7199ddaa24 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88040074e800, cur 1537583451 expire 1537583301 last 1537583224 Sep 21 19:32:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:33:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2865f4c-4aa3-54d8-27ba-84b5a2a76379 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d5d86c00, cur 1537583638 expire 1537583488 last 1537583411 Sep 21 19:39:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 44fe6f01-1896-e078-85c9-03d0bbd34616 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185d7d5c00, cur 1537583950 expire 1537583800 last 1537583723 Sep 21 19:41:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:44:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:45:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0421c91e-378e-d0cb-ae82-8060a5f550d9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031f763000, cur 1537584317 expire 1537584167 last 1537584090 Sep 21 19:49:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:50:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5a37ecf5-e067-eb61-2d9d-56d0ff7ebfa9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880675bfc400, cur 1537584621 expire 1537584471 last 1537584394 Sep 21 19:52:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:53:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e159851b-c196-ee9a-1c15-827974f36485 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015ca7000, cur 1537584816 expire 1537584666 last 1537584589 Sep 21 19:55:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 19:56:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 95fe868f-5393-a69b-3058-ec20afd18230 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c93ff6c00, cur 1537585003 expire 1537584853 last 1537584776 Sep 21 20:01:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 20:02:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b02ed63-094d-a4f5-7290-edcf363e722d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88059c939800, cur 1537585379 expire 1537585229 last 1537585152 Sep 21 20:08:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 20:09:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 31bdefd3-5954-deb5-71b2-d2b76547ad57 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa8bb9800, cur 1537585796 expire 1537585646 last 1537585569 Sep 21 20:15:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 21 20:16:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2201662d-1576-ef43-0a29-5c9c66e84fb0 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc3dc00, cur 1537586167 expire 1537586017 last 1537585940 Sep 21 21:08:45 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 21 21:28:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d71c28eb-1823-1c2b-34cd-01b68111778f (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f19cb2c00, cur 1537590492 expire 1537590342 last 1537590265 Sep 21 21:28:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 21 21:50:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 21 22:36:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e4fbb447-ecb2-fcf3-831a-4004e8390a2f (at 10.9.113.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea512a400, cur 1537594610 expire 1537594460 last 1537594383 Sep 21 22:56:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e4fbb447-ecb2-fcf3-831a-4004e8390a2f (at 10.9.113.12@o2ib4) Sep 22 04:12:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49469e14-9b94-cb64-8a80-f930b727d984 (at 10.9.115.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85a8ac00, cur 1537614778 expire 1537614628 last 1537614551 Sep 22 04:33:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49469e14-9b94-cb64-8a80-f930b727d984 (at 10.9.115.1@o2ib4) Sep 22 10:18:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 764eb92c-8c36-fc54-b1d8-c70cd6139b0c (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fbef66c00, cur 1537636738 expire 1537636588 last 1537636511 Sep 22 10:41:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 22 11:53:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9fe79be8-4a4e-95b8-f02b-6ce1ca8a15b2 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c6850c00, cur 1537642411 expire 1537642261 last 1537642184 Sep 22 12:20:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Sep 22 12:47:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Sep 22 12:47:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0d3d5147-0e76-0f8e-f14f-c72e0dd5ecd4 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001a2af800, cur 1537645653 expire 1537645503 last 1537645426 Sep 22 14:05:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec4b3e77-b108-1fb2-cfd7-f0df1715ca6c (at 10.8.2.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7b97400, cur 1537650300 expire 1537650150 last 1537650073 Sep 22 14:18:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b676dbd6-53c4-0df6-4a30-159889a2b255 (at 10.9.114.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bef20b800, cur 1537651080 expire 1537650930 last 1537650853 Sep 22 14:18:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 22 14:21:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 507e1289-cf91-2386-05ae-b29860d205c8 (at 10.9.114.9@o2ib4) Sep 22 14:34:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb5dc300-0aa3-7343-af8c-d6fe8a188a6f (at 10.9.114.15@o2ib4) Sep 22 14:49:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 34e2e2bc-8c36-fd17-beee-29c564c8b6ee (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e8242800, cur 1537652984 expire 1537652834 last 1537652757 Sep 22 15:12:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 25a1eb8e-cba2-6caa-31b6-a6e02afe3dfb (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003cfb3800, cur 1537654331 expire 1537654181 last 1537654104 Sep 22 15:12:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Sep 22 15:48:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f98dfbf8-4bf5-aad9-2b0b-55f76ab36c18 (at 10.9.115.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880480ea5400, cur 1537656513 expire 1537656363 last 1537656286 Sep 22 16:03:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 989d2e8f-13ec-a59c-a9bd-e2c728310cf1 (at 10.9.115.5@o2ib4) Sep 22 16:15:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d82600f4-f1e6-aa06-fb5a-dba42dcdff94 (at 10.8.20.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808f53e9400, cur 1537658137 expire 1537657987 last 1537657910 Sep 22 16:15:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Sep 22 17:49:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e37fecd3-1f0f-58f5-2d3b-ef0ed4e1a65f (at 10.12.4.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be66000, cur 1537663747 expire 1537663597 last 1537663520 Sep 22 17:59:06 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 22 18:14:18 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 22 18:14:26 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 22 18:14:28 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 22 18:48:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b165274-a69a-fb60-7573-7316b6f2719b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a72f58c00, cur 1537667319 expire 1537667169 last 1537667092 Sep 22 18:49:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 18:58:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 18:59:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fbb9dd0d-2c3b-647e-73db-03596b19818e (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809dc2a2000, cur 1537667944 expire 1537667794 last 1537667717 Sep 22 19:08:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 19:09:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f808f955-4452-7f6f-0f34-494522e6f1d1 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fb446000, cur 1537668553 expire 1537668403 last 1537668326 Sep 22 19:13:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5b616874-96b4-d867-c816-371fa195cb61 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e17d99000, cur 1537668794 expire 1537668644 last 1537668567 Sep 22 19:15:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 19:36:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 633a51fa-3606-b746-16fc-c0ddd3a0bd2d (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e902bac00, cur 1537670191 expire 1537670041 last 1537669964 Sep 22 19:43:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 19:44:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5fd90079-1b2c-5646-6cfb-3f2edddf6b8f (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880751707800, cur 1537670658 expire 1537670508 last 1537670431 Sep 22 19:54:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 19:55:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6fcdf6c-a33e-0be5-c860-278f23f653af (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802ec181400, cur 1537671338 expire 1537671188 last 1537671111 Sep 22 19:56:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 22 20:02:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 20:03:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38009a86-7e16-3b6e-5d2a-b584bd4aac24 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2111000, cur 1537671814 expire 1537671664 last 1537671587 Sep 22 20:09:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 20:10:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9a7ee1f6-bf8b-af0d-8a30-b7e767f05de5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c27d7c00, cur 1537672227 expire 1537672077 last 1537672000 Sep 22 20:17:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 20:17:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b3e811ce-4f0d-8b16-7d84-9e1f62209f1b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880751700000, cur 1537672629 expire 1537672479 last 1537672402 Sep 22 20:24:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 20:25:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cf8cfa9d-7ace-770f-0b24-eedefb10e465 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d288bcc00, cur 1537673103 expire 1537672953 last 1537672876 Sep 22 20:27:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 20:28:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a96633a8-9911-910d-cb84-f3c34d2978f9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802ec181c00, cur 1537673293 expire 1537673143 last 1537673066 Sep 22 20:40:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 20:40:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b2ee7964-c2e9-2fb4-7616-882389c8721a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805ed130000, cur 1537674035 expire 1537673885 last 1537673808 Sep 22 20:51:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 22 20:51:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 73ec5403-8435-06e3-facd-8604ad1ae47d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eac4da800, cur 1537674707 expire 1537674557 last 1537674480 Sep 22 21:47:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e37fecd3-1f0f-58f5-2d3b-ef0ed4e1a65f (at 10.12.4.41@o2ib) Sep 22 22:38:24 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 23 03:09:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 09db39ab-ab74-c18d-09a2-ed0d89f6642c (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c46165c00, cur 1537697373 expire 1537697223 last 1537697146 Sep 23 03:24:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 23 03:25:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ed9649d-f83a-3d55-993d-4ac14450cbc7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014c050800, cur 1537698323 expire 1537698173 last 1537698096 Sep 23 03:32:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ecdcc51-1a4e-afc8-db9b-35bd8ca6b7c8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88029d8e9c00, cur 1537698770 expire 1537698620 last 1537698543 Sep 23 03:33:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 23 03:36:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d85dd6b-c92b-660d-fcaa-6da2163a1f77 (at 10.8.1.14@o2ib6) Sep 23 03:40:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 939cb044-990f-a2fe-4467-632dc886400d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210d000, cur 1537699244 expire 1537699094 last 1537699017 Sep 23 03:40:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 03:42:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 23 03:44:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d8a4134b-9222-884b-12af-929ed7278746 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808d6af3400, cur 1537699483 expire 1537699333 last 1537699256 Sep 23 03:49:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 03:50:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e71513c9-9006-c6f5-de7a-a90dd0dab074 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f196e2800, cur 1537699822 expire 1537699672 last 1537699595 Sep 23 03:53:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e85e50e-a01e-b52d-5410-5062e58e23e5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda288400, cur 1537699988 expire 1537699838 last 1537699761 Sep 23 04:04:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:04:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 23 04:07:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:07:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3aa0b9d6-3790-c0c2-ab99-babd5d99c6c9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4cc34800, cur 1537700871 expire 1537700721 last 1537700644 Sep 23 04:07:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 23 04:10:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:10:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e79a429-471d-4afb-7f52-104b97219b26 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b99267000, cur 1537701050 expire 1537700900 last 1537700823 Sep 23 04:14:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:14:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55c78359-7002-2586-b960-e01dfa47b2bc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8657bc00, cur 1537701291 expire 1537701141 last 1537701064 Sep 23 04:18:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:19:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3ae3482f-f655-4f07-3f21-ca5039f86ff3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c62ff000, cur 1537701548 expire 1537701398 last 1537701321 Sep 23 04:42:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c2fe2ac5-f2be-95c7-2cfe-db688553ab2e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880074ea9c00, cur 1537702944 expire 1537702794 last 1537702717 Sep 23 04:43:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:47:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:47:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb28174e-e4d2-e5d8-31a7-1fe7b93146a1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804b9f8e400, cur 1537703268 expire 1537703118 last 1537703041 Sep 23 04:51:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:51:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fb810064-f304-341b-1108-a66974051c92 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e3c5bc00, cur 1537703496 expire 1537703346 last 1537703269 Sep 23 04:55:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 04:55:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74e80fd6-7d8a-9c02-95fe-a4c1b7abbe63 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2115400, cur 1537703737 expire 1537703587 last 1537703510 Sep 23 05:12:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f07c8f9-ca1d-8bff-d789-0b271f7cd5ad (at 10.12.4.43@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e20c70c00, cur 1537704745 expire 1537704595 last 1537704518 Sep 23 05:13:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca21d097-a095-ce1a-3050-8700484deba8 (at 10.12.4.43@o2ib) Sep 23 05:15:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 05:15:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 855cc1dd-4657-1dac-6992-2c23205fe489 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8dc60c00, cur 1537704951 expire 1537704801 last 1537704724 Sep 23 05:19:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 05:20:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90b0c3d0-090c-d226-f80b-bc13f0f557c4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803dd660400, cur 1537705230 expire 1537705080 last 1537705003 Sep 23 08:28:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 506cacc5-77a1-15fc-9357-90867c00be88 (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802fde29000, cur 1537716487 expire 1537716337 last 1537716260 Sep 23 08:52:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 08:52:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2cd3b803-229c-bbe6-d794-76d0cb122f71 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd65b800, cur 1537717945 expire 1537717795 last 1537717718 Sep 23 09:00:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 09:01:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a1560fb7-ebb5-610f-9db0-ad2619bf1dbe (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc48ce000, cur 1537718484 expire 1537718334 last 1537718257 Sep 23 09:47:41 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 23 09:47:41 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 23 10:31:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 182c7630-1635-3231-a47b-7176eb1d6df4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7bea000, cur 1537723890 expire 1537723740 last 1537723663 Sep 23 10:35:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 10:39:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 19135e23-19c3-9b44-4fcc-f855cac64a8f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afa400, cur 1537724392 expire 1537724242 last 1537724165 Sep 23 10:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 11:40:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 787ee8fa-ae56-1906-0377-c9773930f9b2 (at 10.9.115.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f62571800, cur 1537728008 expire 1537727858 last 1537727781 Sep 23 11:57:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 24e7587a-1582-e0c6-527f-40973b275583 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880730188400, cur 1537729073 expire 1537728923 last 1537728846 Sep 23 11:58:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Sep 23 12:00:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 32b9b8b9-b19c-790e-c493-0374cb677e13 (at 10.9.115.2@o2ib4) Sep 23 12:38:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e29a1c0-a93e-c888-3195-138ceb674a56 (at 10.8.6.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68fc00, cur 1537731512 expire 1537731362 last 1537731285 Sep 23 12:39:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1db58f68-7c4f-2629-6fce-82554b89fc05 (at 10.8.6.9@o2ib6) Sep 23 12:42:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 12:46:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 12:47:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d301622-fa38-ad04-f77c-9d219247638d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880010d67400, cur 1537732027 expire 1537731877 last 1537731800 Sep 23 12:47:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 23 12:51:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 12:52:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e09cdcc-842c-cb09-8c25-5a89df3584e1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808fa264400, cur 1537732338 expire 1537732188 last 1537732111 Sep 23 13:00:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b689373a-251e-396c-9c64-52ea8ef6d178 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817eaf07000, cur 1537732845 expire 1537732695 last 1537732618 Sep 23 13:04:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 13:09:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 13:09:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 72366be5-45e6-0b37-ee58-e6e3e7820dfb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e3800, cur 1537733395 expire 1537733245 last 1537733168 Sep 23 13:13:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 13:13:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd0e4ade-c742-dd74-15f5-44550fde3a8e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053efa000, cur 1537733610 expire 1537733460 last 1537733383 Sep 23 13:16:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 13:17:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79342a7b-1572-e27a-0184-29c08240ba82 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801fc9d6800, cur 1537733855 expire 1537733705 last 1537733628 Sep 23 13:21:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e34c607-2fa2-45f1-4f57-03a2d93a78fd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11865000, cur 1537734080 expire 1537733930 last 1537733853 Sep 23 13:22:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 13:25:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 13:26:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b683c296-6f58-0da2-2494-bf04cb0530a3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2111400, cur 1537734383 expire 1537734233 last 1537734156 Sep 23 16:29:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cfb68c30-ab78-dd4f-da36-e9ec9e167d7e (at 10.9.112.5@o2ib4) Sep 23 16:30:17 oak-md1-s2 kernel: LustreError: 10559:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.112.5@o2ib4) returned error from blocking AST (req@ffff880abde5bf00 x1610628042839824 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881a72e70a00/0x6c00c3efa3b360cc lrc: 4/0,0 mode: PR/PR res: [0x200014a42:0xde1c:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.9.112.5@o2ib4 remote: 0xa5a5f3a7e3dd1665 expref: 10191 pid: 10673 timeout: 6029760522 lvb_type: 0 Sep 23 16:30:17 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.112.5@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Sep 23 16:38:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 636f082f-5ef7-6fe4-ced5-2e1dad95d8f9 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93132800, cur 1537745902 expire 1537745752 last 1537745675 Sep 23 16:59:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 986bf173-8a48-8326-0b87-c02bc7db3b53 (at 10.8.9.8@o2ib6) Sep 23 19:39:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cfb68c30-ab78-dd4f-da36-e9ec9e167d7e (at 10.9.112.5@o2ib4) Sep 23 19:40:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 629cc5a9-f144-d97b-a878-a2c9ae5cb9af (at 10.9.112.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880526a7c000, cur 1537756841 expire 1537756691 last 1537756614 Sep 23 20:02:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8d15b20b-cce0-3a08-fe85-16ffc43b17df (at 10.8.10.1@o2ib6) Sep 23 20:02:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 032c5da9-e230-2343-2e08-45a3dbc63be1 (at 10.8.10.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fddf49000, cur 1537758166 expire 1537758016 last 1537757939 Sep 23 20:16:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 20:17:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f253eeb-068d-ba96-4e49-164aa505ef59 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11867400, cur 1537759025 expire 1537758875 last 1537758798 Sep 23 20:26:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2cf8c48-7875-a0ce-44e5-e4dcee52b93f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3af8800, cur 1537759610 expire 1537759460 last 1537759383 Sep 23 20:27:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 20:37:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 20:37:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cf2c0817-0259-4f70-0182-d0e36010a468 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9d400, cur 1537760269 expire 1537760119 last 1537760042 Sep 23 23:58:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 23 23:58:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ee8495f5-e651-233e-9eee-47f6d0b53247 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9d000, cur 1537772332 expire 1537772182 last 1537772105 Sep 24 00:37:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 00:38:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e41f99a-c3a7-897b-ea0d-041331bdd83d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272370c00, cur 1537774685 expire 1537774535 last 1537774458 Sep 24 01:23:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 01:23:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65d706be-3a5d-db77-45bc-84aab5be5b30 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880adae72000, cur 1537777399 expire 1537777249 last 1537777172 Sep 24 01:26:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20f936c6-8057-f8ee-0251-ee067bc36b40 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88076f788800, cur 1537777611 expire 1537777461 last 1537777384 Sep 24 02:07:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aa81e933-816d-edb4-a86d-a0522ce2869c (at 10.12.4.82@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be01400, cur 1537780062 expire 1537779912 last 1537779835 Sep 24 05:15:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 05:30:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 233d2115-9a34-6488-9949-694162d69282 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880507633400, cur 1537792241 expire 1537792091 last 1537792014 Sep 24 05:31:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 05:39:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 05:39:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 603ab400-0ab0-1177-f3f1-05b334dd6c6d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804234b1000, cur 1537792793 expire 1537792643 last 1537792566 Sep 24 05:49:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bdf01986-4991-b29e-19d9-6eefc6ca4c8b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880de3af5c00, cur 1537793352 expire 1537793202 last 1537793125 Sep 24 06:15:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 06:20:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 06:20:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d668e16f-0426-d910-abc0-af2029420e20 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88037d285c00, cur 1537795252 expire 1537795102 last 1537795025 Sep 24 06:27:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 06:28:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9794073-4990-0b0e-92f1-6b62433eedcc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5fcf6000, cur 1537795708 expire 1537795558 last 1537795481 Sep 24 06:33:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 06:33:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1592a5ad-f4b3-a3c8-bd03-d8ac91074fcf (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b7eafdc00, cur 1537796028 expire 1537795878 last 1537795801 Sep 24 06:38:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cfb68c30-ab78-dd4f-da36-e9ec9e167d7e (at 10.9.112.5@o2ib4) Sep 24 06:39:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b2264601-3fdd-3877-38da-94484b95db6c (at 10.9.112.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc230b000, cur 1537796380 expire 1537796230 last 1537796153 Sep 24 06:39:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 06:46:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 06:46:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7777e513-e570-19d7-2055-af3196f11966 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1733e800, cur 1537796800 expire 1537796650 last 1537796573 Sep 24 06:46:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 06:54:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 06:55:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9be7da3b-eaab-b996-80d8-8cefff211e5c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88022a077000, cur 1537797313 expire 1537797163 last 1537797086 Sep 24 07:01:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 07:02:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d360955-f319-105a-b63f-61b7c1849e8a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88092a98e000, cur 1537797721 expire 1537797571 last 1537797494 Sep 24 07:07:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 07:08:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b21686f8-ce20-9b72-005f-1e8aed9a902c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a3bec2800, cur 1537798089 expire 1537797939 last 1537797862 Sep 24 07:13:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 07:13:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e164f91-1542-7b6f-9c1d-06346549b655 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800597b7000, cur 1537798431 expire 1537798281 last 1537798204 Sep 24 07:27:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 07:27:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aab043ec-1cec-5330-7bf3-142e605fac9b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fddf7d000, cur 1537799268 expire 1537799118 last 1537799041 Sep 24 07:33:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71dfdcef-d786-5cfc-d840-af4faa251a0f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803f0e31400, cur 1537799632 expire 1537799482 last 1537799405 Sep 24 07:41:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 07:41:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 07:41:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71b258af-17c2-1d95-03b7-8c0d1ee64619 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef2883000, cur 1537800092 expire 1537799942 last 1537799865 Sep 24 08:35:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 08:36:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30a3a76c-7bf7-44e3-84b5-c040f073013d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea36fc800, cur 1537803371 expire 1537803221 last 1537803144 Sep 24 08:42:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 08:42:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f097e70f-a5dd-bd23-9227-21ea9fd6bb60 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f11b000, cur 1537803769 expire 1537803619 last 1537803542 Sep 24 08:47:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 08:48:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bda7cd3d-cfcb-8440-7950-ddd86379f3cf (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a4cf50c00, cur 1537804111 expire 1537803961 last 1537803884 Sep 24 08:53:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 08:53:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e988dff5-ec80-19aa-dd8a-f12d3b33a2d9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b1ec0f800, cur 1537804421 expire 1537804271 last 1537804194 Sep 24 09:04:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 119c2d88-d272-12e5-6d4b-a0384fb9d18b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab3ef1400, cur 1537805078 expire 1537804928 last 1537804851 Sep 24 09:14:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 09:20:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 09:20:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 41091039-cc92-98dc-419d-70c9e3fbc803 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6cc6c400, cur 1537806024 expire 1537805874 last 1537805797 Sep 24 09:21:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 032bd5ed-ed2a-fdf9-f3ac-3c9ba252a3f5 (at 10.12.4.73@o2ib) in 159 seconds. I think it's dead, and I am evicting it. exp ffff8810391e1c00, cur 1537806100 expire 1537805950 last 1537805941 Sep 24 09:23:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 032bd5ed-ed2a-fdf9-f3ac-3c9ba252a3f5 (at 10.12.4.73@o2ib) Sep 24 09:37:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 23ee85d3-4440-4886-cf3c-4a701ac57671 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88053d6ca000, cur 1537807056 expire 1537806906 last 1537806829 Sep 24 09:38:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Sep 24 09:59:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79ccde86-b2e7-6f23-1ca5-72f6ada5029d (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080dbc7000, cur 1537808343 expire 1537808193 last 1537808116 Sep 24 10:00:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 21da025e-da3c-8e86-351d-56e74c5d568e (at 10.8.17.26@o2ib6) in 181 seconds. I think it's dead, and I am evicting it. exp ffff8805c27d2400, cur 1537808419 expire 1537808269 last 1537808238 Sep 24 10:00:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 10:04:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1bae682b-93dc-3120-1279-ce37c38d925c (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c42734000, cur 1537808675 expire 1537808525 last 1537808448 Sep 24 10:08:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 10:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac735f8b-e28f-67ef-de4a-9d3602939ed3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf278000, cur 1537808942 expire 1537808792 last 1537808715 Sep 24 10:15:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 10:15:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 122444c4-68c3-8adc-51b8-916293882c1d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802da796000, cur 1537809340 expire 1537809190 last 1537809113 Sep 24 10:21:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 10:21:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c247a2a4-bb8c-73c0-f0b5-737aa384a966 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2d630c00, cur 1537809686 expire 1537809536 last 1537809459 Sep 24 10:27:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 32ee804c-a8a6-0468-a932-4d9175221dc4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0a6f8400, cur 1537810067 expire 1537809917 last 1537809840 Sep 24 10:35:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8e073658-5d71-c10a-9d31-1658501c853d (at 10.9.107.27@o2ib4) Sep 24 10:35:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 10:36:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a352148b-5800-389d-16c1-750f8ac395e9 (at 10.9.107.27@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d3800, cur 1537810616 expire 1537810466 last 1537810389 Sep 24 10:42:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 64c97e86-a341-6474-e6ee-5f677d8179d9 (at 10.9.107.27@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacc25800, cur 1537810944 expire 1537810794 last 1537810717 Sep 24 10:46:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 10:46:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 10:46:57 oak-md1-s2 kernel: LNet: 3780:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 24 10:47:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 11:03:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Sep 24 11:03:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 11:10:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8a406b01-c1f1-13d3-2be2-8842e2875d1d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fb5c4c00, cur 1537812616 expire 1537812466 last 1537812389 Sep 24 11:15:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 11:15:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 11:16:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c941516-5275-f580-332b-6032872eb810 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88048a9ee000, cur 1537812970 expire 1537812820 last 1537812743 Sep 24 11:29:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 24 11:30:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 11b85240-480e-af95-3617-4c41d2ed6e5b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7bef000, cur 1537813813 expire 1537813663 last 1537813586 Sep 24 11:37:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b226b045-03db-a112-8a50-1c0619375b58 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880761e30800, cur 1537814230 expire 1537814080 last 1537814003 Sep 24 11:43:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 24 11:43:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 11:43:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aa11beb6-c5fc-0a7c-e4b6-6ea8d4fc1a38 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a304bd800, cur 1537814638 expire 1537814488 last 1537814411 Sep 24 11:47:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dedc4554-b3f0-f879-4c9b-d36f49346aa9 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880730189c00, cur 1537814840 expire 1537814690 last 1537814613 Sep 24 11:50:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9fea059e-37d6-9fac-68f4-c079886844ac (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd64a400, cur 1537815013 expire 1537814863 last 1537814786 Sep 24 11:53:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 11:53:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 11:53:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 11:53:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 11:57:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7dc1dab4-beef-fd47-fce3-9927b73c62ac (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11864400, cur 1537815479 expire 1537815329 last 1537815252 Sep 24 12:01:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 456710b4-e54c-28bd-6119-82d2c1442af2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806e8cbe000, cur 1537815713 expire 1537815563 last 1537815486 Sep 24 12:01:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 12:06:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 12:06:05 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 24 12:06:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e373313-bd6f-cc2d-06cc-02a1862bc049 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805ed131c00, cur 1537816019 expire 1537815869 last 1537815792 Sep 24 12:15:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 62fcf54d-65ac-9322-aded-571c4de05b09 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8dc65000, cur 1537816551 expire 1537816401 last 1537816324 Sep 24 12:15:51 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 24 12:19:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 12:19:01 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 24 12:28:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d3b2d10-3d89-321c-1692-1ac5f16dfe72 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0e1e4c00, cur 1537817328 expire 1537817178 last 1537817101 Sep 24 12:28:48 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 12:31:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 12:31:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 12:48:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 12:48:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 12:49:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1f06b77c-3976-8769-12b8-e4eca89abca6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881827a77400, cur 1537818541 expire 1537818391 last 1537818314 Sep 24 12:49:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 12:57:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e1eff494-8fab-bea2-ee55-95c142ab15e6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800280d4400, cur 1537819039 expire 1537818889 last 1537818812 Sep 24 13:02:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 13:02:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 13:03:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f27936b-d8c5-cd3c-61c2-d9fbce911af8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005a0cb000, cur 1537819381 expire 1537819231 last 1537819154 Sep 24 13:03:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 13:09:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 86230e64-4dd9-a0ff-3514-ea8e8097e13a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014c057800, cur 1537819764 expire 1537819614 last 1537819537 Sep 24 13:09:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 13:14:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 13:14:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 13:25:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b1a8197e-21cb-faf4-567e-fc09af6c3629 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880814641c00, cur 1537820743 expire 1537820593 last 1537820516 Sep 24 13:25:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 13:26:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Sep 24 13:26:41 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Sep 24 13:35:34 oak-md1-s2 kernel: LNet: 3780:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 24 13:35:37 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537821326/real 1537821330] req@ffff8801bb832100 x1610628071018496/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537821336 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 13:35:37 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Sep 24 13:35:45 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537821331/real 0] req@ffff88059efa5400 x1610628071020560/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537821345 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 13:35:45 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 24 13:36:01 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537821347/real 1537821347] req@ffff88059efa5400 x1610628071020560/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537821361 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Sep 24 13:36:01 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Sep 24 13:36:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 13:36:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 13:36:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 13:36:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 13:37:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 13:47:42 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537822055/real 0] req@ffff88004363b900 x1610628071823936/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537822062 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 13:47:42 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 24 13:48:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 357c9327-5943-7b6b-bf71-96f316a93e47 (at 10.0.2.247@o2ib5) Sep 24 13:48:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 13:48:07 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537822080/real 1537822080] req@ffff88004363b900 x1610628071823936/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537822087 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 24 13:48:07 oak-md1-s2 kernel: LustreError: 10525:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.0.2.247@o2ib5) returned error from blocking AST (req@ffff88004363b900 x1610628071823936 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff88005a2d0600/0x6c00c3eff3bf1085 lrc: 4/0,0 mode: PR/PR res: [0x200014711:0x163ac:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.0.2.247@o2ib5 remote: 0xf247c97b361a72b6 expref: 8286 pid: 10574 timeout: 6107180251 lvb_type: 0 Sep 24 13:48:07 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.0.2.247@o2ib5 was evicted due to a lock blocking callback time out: rc -107 Sep 24 13:54:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 19515f44-d8d0-285c-c2fe-87c11cad5557 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880756288c00, cur 1537822477 expire 1537822327 last 1537822250 Sep 24 14:02:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 14:02:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 14:02:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 876d6b41-f8d5-205d-485b-b4ebef5090a9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a0400, cur 1537822958 expire 1537822808 last 1537822731 Sep 24 14:07:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9a71fc5-edc6-890f-1dd6-b31141205115 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809dc2a7400, cur 1537823278 expire 1537823128 last 1537823051 Sep 24 14:17:02 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537823815/real 0] req@ffff881534b95100 x1610628072715536/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537823822 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 14:17:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 14:17:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 14:17:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 24 14:18:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 14:40:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9513586-03db-7960-013a-41302fe54a94 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803d1770000, cur 1537825254 expire 1537825104 last 1537825027 Sep 24 14:51:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 24 14:51:18 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 24 14:54:41 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 24 14:55:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5baa8054-5718-a9ff-3cdd-3c667f32855f (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dd12df800, cur 1537826113 expire 1537825963 last 1537825886 Sep 24 15:02:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94caff09-2d7e-050e-e7bb-725d211346da (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880526a7f400, cur 1537826575 expire 1537826425 last 1537826348 Sep 24 15:13:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Sep 24 15:14:03 oak-md1-s2 kernel: Lustre: 10520:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537827236/real 0] req@ffff881840b14b00 x1610628075270944/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537827243 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 15:14:16 oak-md1-s2 kernel: Lustre: 10520:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537827249/real 1537827249] req@ffff881840b14b00 x1610628075270944/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537827256 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Sep 24 15:14:19 oak-md1-s2 kernel: LNet: 3780:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 24 15:14:21 oak-md1-s2 kernel: Lustre: 4322:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537827254/real 1537827260] req@ffff8809a3c11b00 x1610628075279328/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537827261 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 15:14:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 15:14:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 15:15:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 15:15:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 15:17:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Sep 24 15:27:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 167059a8-887d-f282-1560-0fbcdd188dab (at 10.8.13.23@o2ib6) Sep 24 15:30:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 922e74c4-b263-7407-3b34-77f72e750e3c (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca1000, cur 1537828256 expire 1537828106 last 1537828029 Sep 24 15:46:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Sep 24 15:46:55 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 24 15:47:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 24 15:47:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 064f0553-caf7-ffd3-d799-ec3285b5432c (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68e800, cur 1537829269 expire 1537829119 last 1537829042 Sep 24 15:51:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 24 15:51:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d9a6977b-3f0a-fe3f-714f-2f9acbe72ef3 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a56400, cur 1537829514 expire 1537829364 last 1537829287 Sep 24 16:09:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75e2b2f8-e736-c400-558f-a9bff3b30d12 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afa000, cur 1537830570 expire 1537830420 last 1537830343 Sep 24 16:23:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6dbb4f09-4c4f-acca-2971-2111641df3e0 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880046e2a000, cur 1537831402 expire 1537831252 last 1537831175 Sep 24 16:24:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 24 16:25:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Sep 24 16:39:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53ace4c1-2802-3359-bb99-33c26a7027f8 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2a23e800, cur 1537832344 expire 1537832194 last 1537832117 Sep 24 16:41:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 24 16:41:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client af01c4c0-f0fb-8f7a-cab2-0e77a86c746f (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807444e9400, cur 1537832517 expire 1537832367 last 1537832290 Sep 24 16:44:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Sep 24 16:48:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e6a93bfe-5f22-9433-8b1b-a7a64a78c9ee (at 10.8.2.9@o2ib6) Sep 24 16:53:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c26648fa-7ebe-2fb6-6200-13f31b24559d (at 10.8.6.30@o2ib6) Sep 24 16:55:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 195e1218-5993-0214-477f-1c81ab69ad1a (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7c633000, cur 1537833318 expire 1537833168 last 1537833091 Sep 24 16:56:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 24 16:56:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 871c9c1c-c213-7348-f787-bbfe6cc6a234 (at 10.8.6.30@o2ib6) in 175 seconds. I think it's dead, and I am evicting it. exp ffff880730188000, cur 1537833394 expire 1537833244 last 1537833219 Sep 24 16:57:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 24 17:03:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Sep 24 17:15:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 24 17:16:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e5cd311-2a64-fcf0-e4d1-fd7fb88f81d5 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880935781800, cur 1537834571 expire 1537834421 last 1537834344 Sep 24 17:18:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Sep 24 17:22:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Sep 24 17:31:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b866110c-3ee5-2859-d180-3c122a72ba67 (at 10.9.107.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b35317c00, cur 1537835475 expire 1537835325 last 1537835248 Sep 24 17:37:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e6ebe66-e4d5-4b7c-f7f9-3ef3148b08ba (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808d6af7400, cur 1537835860 expire 1537835710 last 1537835633 Sep 24 17:38:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 24 17:40:41 oak-md1-s2 kernel: Lustre: 10571:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537836032/real 1537836033] req@ffff8815db323f00 x1610628087083312/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537836041 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 17:40:42 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537836033/real 1537836034] req@ffff881532bebf00 x1610628087083472/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537836042 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 17:40:43 oak-md1-s2 kernel: Lustre: 4239:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537836034/real 1537836036] req@ffff8813d0e43f00 x1610628087084192/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537836043 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 17:40:43 oak-md1-s2 kernel: Lustre: 4239:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Sep 24 17:40:55 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537836041/real 0] req@ffff8814de261e00 x1610628087086400/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537836055 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 17:40:55 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 24 17:41:12 oak-md1-s2 kernel: Lustre: 10559:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537836058/real 0] req@ffff881a9cf16600 x1610628087092368/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537836072 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 24 17:41:12 oak-md1-s2 kernel: Lustre: 10559:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Sep 24 17:41:24 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537836070/real 1537836070] req@ffff8814de261e00 x1610628087086400/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537836084 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Sep 24 17:41:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 17:41:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 17:42:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 24 17:42:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 24 17:44:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 24 17:45:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a28adf2-d0a3-6497-c2b6-52aeedac2c2f (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dda81c400, cur 1537836350 expire 1537836200 last 1537836123 Sep 24 18:28:19 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 24 19:25:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 14c2b408-abd9-458b-d615-dbec8c96605c (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102eebe000, cur 1537842353 expire 1537842203 last 1537842126 Sep 24 19:32:45 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 24 19:52:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Sep 24 19:52:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ba0e37ec-6e8b-30ca-0b46-6d0aa3baa982 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c872c00, cur 1537843977 expire 1537843827 last 1537843750 Sep 24 20:10:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Sep 24 20:11:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6831d3d4-da7a-3397-3104-ff20dc556265 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88067bedc400, cur 1537845063 expire 1537844913 last 1537844836 Sep 24 20:20:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Sep 24 20:21:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5bd95df-0525-5040-57bd-09fce46ab319 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6719f000, cur 1537845676 expire 1537845526 last 1537845449 Sep 24 20:43:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dab73de4-f461-4203-56a9-168702a1443c (at 10.9.112.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edf2800, cur 1537847015 expire 1537846865 last 1537846788 Sep 24 20:44:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cfb68c30-ab78-dd4f-da36-e9ec9e167d7e (at 10.9.112.5@o2ib4) Sep 24 23:11:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 23:12:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0b48e938-58c3-bfe4-d34b-58078dd5180a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f717400, cur 1537855925 expire 1537855775 last 1537855698 Sep 24 23:15:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 23:16:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 01891473-a379-1232-1e1c-ab8167bd732f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808fa3a1800, cur 1537856187 expire 1537856037 last 1537855960 Sep 24 23:23:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de5b295c-59c0-1b74-c91d-fb6f170e355c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880660e88800, cur 1537856588 expire 1537856438 last 1537856361 Sep 24 23:23:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 23:28:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 23:29:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 425810ea-0955-fbf9-d100-7aa6c55eb8d8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809439de800, cur 1537856970 expire 1537856820 last 1537856743 Sep 24 23:35:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 24 23:36:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0f0e091a-e483-f7f6-d50e-004e7f1d6e90 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee9ff2000, cur 1537857365 expire 1537857215 last 1537857138 Sep 25 03:07:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 58ce8883-7735-a192-f3f1-1feb19954296 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0e1e7000, cur 1537870052 expire 1537869902 last 1537869825 Sep 25 03:07:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:11:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:12:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a76a3842-0d75-60d4-0f84-c3d7e7befaab (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f0c38d800, cur 1537870348 expire 1537870198 last 1537870121 Sep 25 03:17:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eafb24f4-2a7d-b5fa-68cc-fb8e27677b2f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f0c353c00, cur 1537870631 expire 1537870481 last 1537870404 Sep 25 03:21:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:25:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:25:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 57d1fcd5-e72e-f71b-d99e-6f916f4d1fd9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802a9bbcc00, cur 1537871122 expire 1537870972 last 1537870895 Sep 25 03:29:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 787b6122-c471-4715-c1dd-fcc730c23bb8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880036e51000, cur 1537871385 expire 1537871235 last 1537871158 Sep 25 03:31:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:34:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:34:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e351b361-280b-7689-b58c-5d11d2dba54a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88903400, cur 1537871697 expire 1537871547 last 1537871470 Sep 25 03:36:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4966203-761f-6dc5-9911-bf9e91bbb908 (at 10.8.13.24@o2ib6) Sep 25 03:38:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d71e393d-4124-d828-83db-d38e52af65f0 (at 10.8.13.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807f58da000, cur 1537871919 expire 1537871769 last 1537871692 Sep 25 03:38:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 03:41:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3eb2a272-3d6f-6e47-4ba6-d047462c12e0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039f59b000, cur 1537872101 expire 1537871951 last 1537871874 Sep 25 03:53:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:57:18 oak-md1-s2 kernel: Lustre: 10484:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537873031/real 1537873034] req@ffff880a5f5f8600 x1610628106695056/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537873038 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Sep 25 03:57:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9a264a7e-5587-5527-5106-50af6a18830c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809c7568800, cur 1537873045 expire 1537872895 last 1537872818 Sep 25 03:57:25 oak-md1-s2 kernel: Lustre: 10484:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537873038/real 1537873034] req@ffff880a5f5f8600 x1610628106695056/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537873045 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Sep 25 03:57:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 03:57:31 oak-md1-s2 kernel: Lustre: 10548:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537873044/real 0] req@ffff8809af9fe300 x1610628106700080/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537873051 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 25 03:57:59 oak-md1-s2 kernel: Lustre: 10484:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537873071/real 1537873071] req@ffff880a5f5f8600 x1610628106695056/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537873078 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Sep 25 03:57:59 oak-md1-s2 kernel: Lustre: 10484:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 25 03:58:02 oak-md1-s2 kernel: LNet: 3779:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 25 03:58:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 25 03:58:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 25 03:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 25 03:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 25 03:59:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 25 04:04:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38a49a52-ef61-2c0a-6d28-32f49a432ba3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d6e215000, cur 1537873473 expire 1537873323 last 1537873246 Sep 25 04:05:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 04:05:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 04:05:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f0b68af4-3f50-6e2d-8409-c2424d5ff765 (at 10.8.4.15@o2ib6) in 156 seconds. I think it's dead, and I am evicting it. exp ffff880a978f9c00, cur 1537873549 expire 1537873399 last 1537873393 Sep 25 04:07:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 468934c8-bd25-066c-7ba5-fc8ddaac13d0 (at 10.12.4.80@o2ib) in 177 seconds. I think it's dead, and I am evicting it. exp ffff8810302f4c00, cur 1537873625 expire 1537873475 last 1537873448 Sep 25 04:12:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cd2cc4e0-478b-6240-ac84-aa3d507bb07f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807acec8400, cur 1537873924 expire 1537873774 last 1537873697 Sep 25 04:13:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 04:17:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0dfff4e5-5e19-de02-f53c-a607d6e830b0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88159f2da000, cur 1537874221 expire 1537874071 last 1537873994 Sep 25 04:20:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 04:20:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 04:32:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 218e03f9-a19c-3cc2-06a8-ca83cbfd69cc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807f58dd400, cur 1537875142 expire 1537874992 last 1537874915 Sep 25 04:32:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 04:36:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 04:49:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 04:59:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 05:00:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e084cdc0-fe95-96ca-1ec0-992fd8deeb45 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a51478400, cur 1537876827 expire 1537876677 last 1537876600 Sep 25 05:00:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 07:29:20 oak-md1-s2 kernel: LNet: 3781:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 25 07:29:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 25 07:29:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 25 07:29:57 oak-md1-s2 kernel: Lustre: 10668:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1537885790/real 0] req@ffff8815eb69bc00 x1610628111393840/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537885797 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 25 07:30:21 oak-md1-s2 kernel: LNet: 3781:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 25 07:30:23 oak-md1-s2 kernel: Lustre: 10668:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537885816/real 1537885816] req@ffff8815eb69bc00 x1610628111393840/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537885823 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Sep 25 07:30:30 oak-md1-s2 kernel: Lustre: 10668:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1537885823/real 1537885829] req@ffff8815eb69bc00 x1610628111393840/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1537885830 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Sep 25 07:30:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 25 08:40:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ba33dd0-01d9-3977-56ea-b5ee97a1ea4a (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b322e0c00, cur 1537890035 expire 1537889885 last 1537889808 Sep 25 10:50:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Sep 25 10:50:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 10:58:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b1abf8a-df96-a927-e1e3-9a6829c09558 (at 10.8.14.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0da8a000, cur 1537898303 expire 1537898153 last 1537898076 Sep 25 10:58:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Sep 25 11:00:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a816ffb-848a-a9f8-e400-c8e354652630 (at 10.8.14.7@o2ib6) Sep 25 11:00:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 11:04:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 25 11:09:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cdcd05c3-54f5-8748-3ca6-a470bb0610fb (at 10.8.4.9@o2ib6) Sep 25 11:09:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 11:16:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 11:16:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 50adfb14-e8a8-cd3a-f48d-c498eb8c7127 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a6400, cur 1537899400 expire 1537899250 last 1537899173 Sep 25 11:41:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 11:41:37 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Sep 25 11:42:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4eba9820-67cc-f24a-3a66-556807387950 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003e694400, cur 1537900926 expire 1537900776 last 1537900699 Sep 25 11:44:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 11:45:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0253db74-540c-95c2-8167-b04947417aa5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805102f4800, cur 1537901133 expire 1537900983 last 1537900906 Sep 25 11:48:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bc0ed84-8f48-2993-e529-2a92fac09f00 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edfb000, cur 1537901332 expire 1537901182 last 1537901105 Sep 25 11:52:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 11:52:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c724d469-acfa-486d-17de-4f4e569f40b9 (at 10.9.107.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810086e8000, cur 1537901564 expire 1537901414 last 1537901337 Sep 25 11:55:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 77f4bc87-fc96-8ea2-5a75-a76e0d560061 (at 10.8.9.7@o2ib6) Sep 25 11:56:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9444497e-8c9b-f230-11b9-1763045e5e7a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2cd400, cur 1537901809 expire 1537901659 last 1537901582 Sep 25 12:00:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 12:00:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 12:01:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7eeb6fb7-cc7b-1456-822c-07e312cf19fa (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880995a95000, cur 1537902076 expire 1537901926 last 1537901849 Sep 25 12:06:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc5a3d40-298c-ff82-8af6-74db919e14a2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804541f2000, cur 1537902373 expire 1537902223 last 1537902146 Sep 25 12:11:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 12:15:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04825f39-17da-dca5-85be-925271d1daca (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582301c00, cur 1537902910 expire 1537902760 last 1537902683 Sep 25 12:28:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Sep 25 12:28:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 25 12:28:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 602e90f4-0914-b0d1-366f-7de711600bb0 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fddf4ac00, cur 1537903692 expire 1537903542 last 1537903465 Sep 25 12:28:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 25 13:06:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ebc8001a-2a0e-33bf-daf5-4dd8dd9eb5ec (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fddf4f800, cur 1537906018 expire 1537905868 last 1537905791 Sep 25 13:07:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 25 14:13:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Sep 25 17:02:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb8ba597-f22e-2cff-40ac-326670441189 (at 10.8.9.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053efa400, cur 1537920139 expire 1537919989 last 1537919912 Sep 25 22:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 25 22:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 25 22:35:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c48c1c07-8a61-d5dd-2326-385527c87f35 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88067bedf400, cur 1537940148 expire 1537939998 last 1537939921 Sep 26 02:43:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client dd37ab47-2674-29f4-6a97-a17dda5b9850 (at 10.8.2.13@o2ib6) reconnecting Sep 26 02:43:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd37ab47-2674-29f4-6a97-a17dda5b9850 (at 10.8.2.13@o2ib6) Sep 26 02:43:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client dd37ab47-2674-29f4-6a97-a17dda5b9850 (at 10.8.2.13@o2ib6) reconnecting Sep 26 02:43:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd37ab47-2674-29f4-6a97-a17dda5b9850 (at 10.8.2.13@o2ib6) Sep 26 06:41:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 87a4f562-9583-0409-06cf-e83a631ccdd3 (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d598a9400, cur 1537969290 expire 1537969140 last 1537969063 Sep 26 08:54:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c45a8775-5e9b-cf77-e376-f5487be20b78 (at 10.8.2.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d3000, cur 1537977271 expire 1537977121 last 1537977044 Sep 26 10:33:59 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 11:25:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c33e18a8-f16d-70f0-c8b9-cfb3d379837d (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041f131c00, cur 1537986331 expire 1537986181 last 1537986104 Sep 26 12:09:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Sep 26 12:12:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0f4c9f7f-0eda-73a1-90a9-d0f0915f58ec (at 10.8.11.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88022a076000, cur 1537989131 expire 1537988981 last 1537988904 Sep 26 13:26:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d909dd1-39fd-681d-8077-aeaf7372fa9a (at 10.8.20.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005a0c8400, cur 1537993598 expire 1537993448 last 1537993371 Sep 26 13:33:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2479713c-d92a-9a4c-31db-188d0ee65bc5 (at 10.8.11.14@o2ib6) Sep 26 13:41:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Sep 26 13:50:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 96f99c4d-7610-8f08-fc43-6b48a6f4ff97 (at 10.8.10.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff3d9cc00, cur 1537995020 expire 1537994870 last 1537994793 Sep 26 13:51:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5f847b31-ead3-0f1f-ad35-9c262bc96b44 (at 10.8.10.34@o2ib6) Sep 26 16:43:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ada8676-2dfb-1860-0a4b-c756a26ba7d3 (at 10.8.9.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee1a04c00, cur 1538005396 expire 1538005246 last 1538005169 Sep 26 16:52:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca4de82a-a28c-8cdd-44d5-f14392933650 (at 10.8.9.1@o2ib6) Sep 26 16:59:19 oak-md1-s2 kernel: Lustre: 10508:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 16:59:53 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 16:59:53 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Sep 26 16:59:59 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 16:59:59 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 26 17:00:46 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:00:46 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Sep 26 17:01:28 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:01:28 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Sep 26 17:01:40 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:01:40 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 24 previous similar messages Sep 26 17:01:57 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:01:57 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Sep 26 17:02:46 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:02:46 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 91 previous similar messages Sep 26 17:04:45 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:04:45 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 72 previous similar messages Sep 26 17:09:03 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:09:03 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 54 previous similar messages Sep 26 17:13:46 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:13:46 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Sep 26 17:24:42 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:24:42 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 148 previous similar messages Sep 26 17:36:36 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:36:36 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 204 previous similar messages Sep 26 17:48:06 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:48:06 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 106 previous similar messages Sep 26 17:59:12 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 17:59:12 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 107 previous similar messages Sep 26 18:11:39 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 18:11:39 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 122 previous similar messages Sep 26 18:28:11 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 18:28:11 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 157 previous similar messages Sep 26 18:38:23 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 18:38:23 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 82 previous similar messages Sep 26 18:55:29 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 26 18:55:29 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 315 previous similar messages Sep 26 19:24:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cab6f2f6-930e-9572-6441-0f5770e53632 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88956400, cur 1538015081 expire 1538014931 last 1538014854 Sep 26 20:36:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Sep 26 20:38:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6085d31b-4f17-dc7a-e944-2b1f39444c50 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc1544c00, cur 1538019503 expire 1538019353 last 1538019276 Sep 26 21:16:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:23:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bd13ddb3-fe52-0aa5-879f-cee39b1f61e3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0f9800, cur 1538022203 expire 1538022053 last 1538021976 Sep 26 21:27:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:30:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:31:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c80fa3cf-08a7-1a5f-0ae8-24c4b5a30df8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee1a04800, cur 1538022687 expire 1538022537 last 1538022460 Sep 26 21:34:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:34:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39a699fb-bdd1-ec3b-645b-19763e942de4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804234b7000, cur 1538022886 expire 1538022736 last 1538022659 Sep 26 21:37:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:38:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e1666f1-3b9c-8ee9-fd3a-901d33e4ac40 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d47485800, cur 1538023090 expire 1538022940 last 1538022863 Sep 26 21:41:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7cfbc6ca-ff0b-9eb3-1ec6-5c8950e6601a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c46161000, cur 1538023284 expire 1538023134 last 1538023057 Sep 26 21:42:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:46:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cca845e7-1a82-47b8-74b0-936483777a3a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c46163400, cur 1538023602 expire 1538023452 last 1538023375 Sep 26 21:55:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:59:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 21:59:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 17898e68-13a3-d802-d22d-b733c5e6d15d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee604e800, cur 1538024393 expire 1538024243 last 1538024166 Sep 26 22:02:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 22:03:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 22cd456c-6691-023e-d9f2-f3e93854a169 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880071c0f800, cur 1538024608 expire 1538024458 last 1538024381 Sep 26 22:06:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 22:06:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 799e6efd-52fd-bbc0-ada8-8d7ca48053fa (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081fe4e400, cur 1538024806 expire 1538024656 last 1538024579 Sep 26 22:09:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 22:10:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8edd9547-a80d-d733-3319-036bc5163817 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a3400, cur 1538025010 expire 1538024860 last 1538024783 Sep 26 22:13:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 22:13:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1f6969fe-103f-65a2-2e9d-c7a86d0319aa (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fb5c0000, cur 1538025227 expire 1538025077 last 1538025000 Sep 26 22:17:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 26 22:17:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9486d34d-4c33-78a0-e18a-f841443ed657 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5bc5c000, cur 1538025462 expire 1538025312 last 1538025235 Sep 26 22:21:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 009266ab-5332-8e88-7918-7fde2edb9e0f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880420552000, cur 1538025697 expire 1538025547 last 1538025470 Sep 26 23:27:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e0419886-b1b5-0ad3-3d90-3e09c4657019 (at 10.8.6.21@o2ib6) reconnecting Sep 26 23:27:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d296a15e-8154-8527-be08-b302d3e42b22 (at 10.8.6.21@o2ib6) Sep 27 01:11:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:14:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:15:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6843b70a-19fa-75ea-b579-188cb64e7f37 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd946000, cur 1538036124 expire 1538035974 last 1538035897 Sep 27 01:18:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8a357f0a-eec3-f59b-7a5b-afb2bdea3bbd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081c231c00, cur 1538036322 expire 1538036172 last 1538036095 Sep 27 01:20:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:24:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59cdddb7-dab8-51fb-de0e-9fc6e9507504 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802a08b3800, cur 1538036650 expire 1538036500 last 1538036423 Sep 27 01:25:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:29:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c0455c9-cd41-3f76-67d9-e46517fd147f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000bd71c00, cur 1538036967 expire 1538036817 last 1538036740 Sep 27 01:29:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:33:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71c43e83-d51e-8d0f-dbd4-ed2eebdef190 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804c26f0800, cur 1538037207 expire 1538037057 last 1538036980 Sep 27 01:33:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:36:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:40:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8806cb8d-d959-ab3e-f090-dc05a4c971ad (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803b5338800, cur 1538037636 expire 1538037486 last 1538037409 Sep 27 01:40:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 01:54:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 01:58:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4197c7a7-e2d5-e8b0-0cb3-67a7cb8d2aa1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809439d8800, cur 1538038708 expire 1538038558 last 1538038481 Sep 27 02:00:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 02:04:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 02:07:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 02:10:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 02:11:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e8d0836-bd8c-2552-a65f-a4203503a906 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68d400, cur 1538039486 expire 1538039336 last 1538039259 Sep 27 02:11:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 02:14:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 02:21:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Sep 27 02:23:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 449c5234-6e14-d209-8359-0f535a0d7301 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802267e5000, cur 1538040195 expire 1538040045 last 1538039968 Sep 27 02:23:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 03:29:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 03:32:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 03:33:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 231861a2-6945-6a87-f58e-82e644e5b8dd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f96a50000, cur 1538044387 expire 1538044237 last 1538044160 Sep 27 03:36:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 03:36:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a14ceab-c05f-59e3-bdc8-25c8eaa7c893 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7de6c000, cur 1538044572 expire 1538044422 last 1538044345 Sep 27 03:39:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c341a657-9c00-f5e4-e7ad-eee38ffde753 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cb32b4000, cur 1538044798 expire 1538044648 last 1538044571 Sep 27 03:48:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 03:48:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 03:51:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53628e25-b823-d9c7-2b92-ac8227aa7ba0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d66e000, cur 1538045518 expire 1538045368 last 1538045291 Sep 27 03:51:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 03:59:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 03:59:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 04:03:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 69d9f17e-4094-2230-56d2-cd914721d03a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88021fc22400, cur 1538046218 expire 1538046068 last 1538045991 Sep 27 04:03:38 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 04:31:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 04:31:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 04:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 04:35:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df874a6a-d691-b4c0-289d-365d94108277 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880343113000, cur 1538048101 expire 1538047951 last 1538047874 Sep 27 04:35:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 04:38:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c7dcef9-8018-8652-ab53-2b96bdfd5508 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880456373800, cur 1538048323 expire 1538048173 last 1538048096 Sep 27 04:39:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 04:43:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2080594-da2f-a457-589e-959f93ab260b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea36fec00, cur 1538048628 expire 1538048478 last 1538048401 Sep 27 04:46:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 04:46:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 04:51:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dfb93460-e27a-f990-4a8e-e5d7cc884a46 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee056dc00, cur 1538049082 expire 1538048932 last 1538048855 Sep 27 04:51:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 04:57:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 04:57:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 05:01:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1768be4b-6ecf-5bf1-2912-fa503987b300 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e3c5cc00, cur 1538049688 expire 1538049538 last 1538049461 Sep 27 05:01:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 05:27:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 05:30:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 05:30:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aa67b816-4e00-39ce-a998-7304f7d6a058 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b28fdf800, cur 1538051458 expire 1538051308 last 1538051231 Sep 27 05:34:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f3a663f7-a3c1-2cd5-c092-c978efb595b9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061e805800, cur 1538051671 expire 1538051521 last 1538051444 Sep 27 05:34:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 05:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7fb83e7a-2fa6-852d-e17c-581c1a573060 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c6853400, cur 1538051911 expire 1538051761 last 1538051684 Sep 27 05:42:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 05:42:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 05:45:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9e72e7a-c0dc-addd-26be-7f3e1adc2f75 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802061c8400, cur 1538052357 expire 1538052207 last 1538052130 Sep 27 05:45:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 05:52:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 05:52:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 05:56:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b4673b7b-18a7-b814-feaa-9fac0241a169 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d18d0d800, cur 1538053009 expire 1538052859 last 1538052782 Sep 27 05:56:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 06:27:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 06:27:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 06:31:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 06:31:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2f2554b-87bb-90cc-cb24-ccb2a788385d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802c8b65000, cur 1538055113 expire 1538054963 last 1538054886 Sep 27 06:31:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 06:35:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 06:35:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4fa09630-c73b-8855-5b4e-28901f625c7d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5fcf0800, cur 1538055327 expire 1538055177 last 1538055100 Sep 27 06:41:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c2101a9f-31ea-4c3e-c7bb-fe61f3ab5bf9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a66cdc800, cur 1538055683 expire 1538055533 last 1538055456 Sep 27 06:42:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 06:46:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cc0d02c0-64b7-b338-fcaf-b66ed4b660a6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d169cb000, cur 1538056000 expire 1538055850 last 1538055773 Sep 27 06:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 06:53:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 06:57:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0007d636-b080-016e-d346-ba9f7f0d4f03 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c6850400, cur 1538056661 expire 1538056511 last 1538056434 Sep 27 06:57:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 07:20:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 07:20:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 07:20:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ad35f39-4cdf-3a4e-55f7-5ecd25179cc7 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88006371ec00, cur 1538058051 expire 1538057901 last 1538057824 Sep 27 07:24:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ffa30599-98fb-1460-3120-246fd18f3bbd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f03e18400, cur 1538058280 expire 1538058130 last 1538058053 Sep 27 07:24:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 07:29:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 07:29:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b45951d1-0600-ef85-1680-963ab66a9f75 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880660e8b800, cur 1538058560 expire 1538058410 last 1538058333 Sep 27 07:36:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5e0e7a5-1930-ddc0-b4b5-c5243f17e5ee (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803b533d400, cur 1538059001 expire 1538058851 last 1538058774 Sep 27 07:36:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 07:43:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 27 07:43:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 27 07:43:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 07:44:48 oak-md1-s2 kernel: LNet: 3779:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 27 07:44:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) reconnecting Sep 27 07:47:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3b201c8-087f-2a30-2361-e2c13384177f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803b533c800, cur 1538059672 expire 1538059522 last 1538059445 Sep 27 07:54:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 07:54:43 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Sep 27 08:00:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 17f0618e-6b4c-61f3-6ac4-ad2af8a2c610 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88029d91b000, cur 1538060410 expire 1538060260 last 1538060183 Sep 27 08:00:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 08:28:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 08:28:38 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 08:43:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client edc3e02b-dae2-c7d5-f8bb-5ab287570fbb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880893868400, cur 1538062996 expire 1538062846 last 1538062769 Sep 27 08:43:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 08:44:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 08:48:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c59f5368-e49a-ce53-3765-46a114319b66 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aeff01c00, cur 1538063281 expire 1538063131 last 1538063054 Sep 27 08:51:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 08:56:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 08:57:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 820dc57a-98e0-9bbf-5459-ff53f4ac9de8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e3e800, cur 1538063825 expire 1538063675 last 1538063598 Sep 27 09:02:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 09:03:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20f03ed0-9afe-4e97-694c-cf472b2228f2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803f0e34800, cur 1538064200 expire 1538064050 last 1538063973 Sep 27 09:03:53 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 09:03:53 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Sep 27 09:08:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 09:12:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 09:16:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e1d6d9d-594c-5f44-9ad9-8d55cb2a04f0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582300c00, cur 1538064967 expire 1538064817 last 1538064740 Sep 27 09:16:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 27 09:25:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 09:25:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 09:29:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0bbbef3-52a8-36d7-00cc-c1697b23c78c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880073b36800, cur 1538065743 expire 1538065593 last 1538065516 Sep 27 09:29:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 09:32:00 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:21:12 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:21:12 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 27 10:21:21 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:21:21 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 27 10:23:34 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:23:34 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 27 10:27:46 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:30:43 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:30:43 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Sep 27 10:31:14 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:31:14 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Sep 27 10:33:09 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:33:09 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 27 10:36:53 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:36:53 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 56 previous similar messages Sep 27 10:40:43 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:40:43 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Sep 27 10:40:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ff0677b8-665a-6aca-62c8-0e6ce35ed2ab (at 10.9.113.4@o2ib4) Sep 27 10:40:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 10:51:29 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 10:51:29 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 108 previous similar messages Sep 27 10:58:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 10:59:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94ee624e-9532-a542-a2d1-120b27894bfd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880727401800, cur 1538071143 expire 1538070993 last 1538070916 Sep 27 11:03:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ad0c5b9-8f2b-96cd-f18b-464a9c52abe1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f2af000, cur 1538071408 expire 1538071258 last 1538071181 Sep 27 11:03:29 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 11:03:29 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 76 previous similar messages Sep 27 11:07:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 11:10:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 11:11:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 315f1c0f-9ad2-9d0e-0fc6-b30068181102 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807b0094c00, cur 1538071871 expire 1538071721 last 1538071644 Sep 27 11:13:31 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 11:13:31 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 113 previous similar messages Sep 27 11:13:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 11:17:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 11:17:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 66fe77bd-84a8-2c84-4332-5ecd48fd51fb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a735fb400, cur 1538072261 expire 1538072111 last 1538072034 Sep 27 11:17:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 11:23:37 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 11:23:37 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 244 previous similar messages Sep 27 11:25:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 11:30:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47693aa3-94e3-1b5f-4730-70890a4fe6cf (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880de3af5400, cur 1538073040 expire 1538072890 last 1538072813 Sep 27 11:30:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 11:33:53 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 11:33:53 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 181 previous similar messages Sep 27 11:36:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 11:44:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d9352d2-17f1-32e2-3050-a81e60d8fbfc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c93ff7800, cur 1538073864 expire 1538073714 last 1538073637 Sep 27 11:44:33 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 11:44:33 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Sep 27 11:47:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 12:01:24 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 12:01:24 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 129 previous similar messages Sep 27 12:03:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 12:03:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 12:03:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b41e266d-0025-ec98-2863-242402a566f7 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804e174cc00, cur 1538075015 expire 1538074865 last 1538074788 Sep 27 12:03:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 12:14:01 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 12:14:01 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 244 previous similar messages Sep 27 12:18:14 oak-md1-s2 kernel: LNet: 361386:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.247@o2ib5 version 12/12 incarnation 1537822079937824/1538075893514415 Sep 27 12:18:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 357c9327-5943-7b6b-bf71-96f316a93e47 (at 10.0.2.247@o2ib5) Sep 27 12:18:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 12:18:44 oak-md1-s2 kernel: LustreError: 10605:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.0.2.247@o2ib5) returned error from blocking AST (req@ffff880fb9f1a100 x1610628350435744 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8806926e1800/0x6c00c3f3814b509e lrc: 4/0,0 mode: PR/PR res: [0x2000172c5:0x9aa:0x0].0x0 bits 0x13 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.0.2.247@o2ib5 remote: 0xbc8b94d40ae0f8d6 expref: 258557 pid: 10552 timeout: 6361017687 lvb_type: 0 Sep 27 12:18:44 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.0.2.247@o2ib5 was evicted due to a lock blocking callback time out: rc -107 Sep 27 12:18:45 oak-md1-s2 kernel: LustreError: 10605:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880fb9f1ef00 x1610628350440704/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 27 12:18:45 oak-md1-s2 kernel: LustreError: 10605:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Sep 27 12:18:58 oak-md1-s2 kernel: LustreError: 10633:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88019afc2100 x1610628350522512/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 27 12:18:58 oak-md1-s2 kernel: LustreError: 10633:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Sep 27 12:24:47 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 12:24:47 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 32 previous similar messages Sep 27 12:32:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 12:33:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2447d1aa-af49-9245-b532-8c06ad5b81cb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809409c0000, cur 1538076789 expire 1538076639 last 1538076562 Sep 27 12:33:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 12:36:09 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 12:36:09 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 182 previous similar messages Sep 27 12:37:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client afc0dfc6-4cf8-dac5-0390-6ed8e510133d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb981ac00, cur 1538077033 expire 1538076883 last 1538076806 Sep 27 12:47:52 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 12:47:52 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Sep 27 12:52:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 12:52:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 12:53:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e5159736-fd36-7ecf-c96f-df18224d5e43 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039234400, cur 1538077982 expire 1538077832 last 1538077755 Sep 27 13:08:10 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 13:08:10 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 43 previous similar messages Sep 27 13:26:40 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 13:26:40 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Sep 27 13:45:36 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 13:45:36 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Sep 27 13:55:20 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 13:55:20 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Sep 27 13:58:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 13:59:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 991ef047-061a-555a-7b1b-fdc062c22c96 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880004435000, cur 1538081954 expire 1538081804 last 1538081727 Sep 27 14:02:05 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:02:05 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Sep 27 14:02:11 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:02:11 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Sep 27 14:06:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 407a414c-c430-3e56-b60b-992a0ad44273 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881463fe5000, cur 1538082373 expire 1538082223 last 1538082146 Sep 27 14:07:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 14:08:23 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:08:23 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Sep 27 14:11:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Sep 27 14:12:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e63551d-5942-dd59-cb00-a24d987beed9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040ed6c00, cur 1538082757 expire 1538082607 last 1538082530 Sep 27 14:16:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d96a4c2-1c63-0847-63d1-29aaafa7e711 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc39c00, cur 1538083004 expire 1538082854 last 1538082777 Sep 27 14:32:14 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:32:14 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 38 previous similar messages Sep 27 14:33:29 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:33:29 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 42 previous similar messages Sep 27 14:35:18 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:35:18 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Sep 27 14:37:03 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:37:03 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 34 previous similar messages Sep 27 14:37:23 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:37:23 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 38 previous similar messages Sep 27 14:39:05 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:39:05 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Sep 27 14:44:32 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:44:32 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Sep 27 14:56:47 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 14:56:47 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 31 previous similar messages Sep 27 15:02:04 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 15:02:04 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Sep 27 15:09:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 522fccbb-92dd-37d5-8b19-c194069b26bb (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eac4de400, cur 1538086172 expire 1538086022 last 1538085945 Sep 27 15:12:06 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 15:12:06 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 103 previous similar messages Sep 27 15:22:06 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 15:22:06 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3626 previous similar messages Sep 27 15:32:23 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 15:32:23 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9401 previous similar messages Sep 27 15:43:03 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 15:43:03 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6581 previous similar messages Sep 27 15:53:11 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 15:53:11 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6713 previous similar messages Sep 27 16:03:11 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 16:03:11 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1621 previous similar messages Sep 27 16:13:15 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 16:13:15 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2592 previous similar messages Sep 27 16:23:25 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 16:23:25 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2139 previous similar messages Sep 27 16:33:53 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 16:33:53 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1974 previous similar messages Sep 27 16:42:22 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.249@o2ib5 version 12/12 incarnation 1536080450534604/1538091740565099 Sep 27 16:42:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14c7ca7b-e54a-24f5-d845-29bab5f1ef8a (at 10.0.2.249@o2ib5) Sep 27 16:44:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 56e30ff8-e3a9-88d1-96bb-8458f1d6a65c (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee9ff2800, cur 1538091842 expire 1538091692 last 1538091615 Sep 27 16:44:06 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 16:44:06 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1177 previous similar messages Sep 27 16:54:32 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 16:54:32 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1470 previous similar messages Sep 27 17:04:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Sep 27 17:04:33 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 17:04:33 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3285 previous similar messages Sep 27 17:15:06 oak-md1-s2 kernel: LNet: 461:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.247@o2ib5 version 12/12 incarnation 1538075893514415/1538093705278552 Sep 27 17:15:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 357c9327-5943-7b6b-bf71-96f316a93e47 (at 10.0.2.247@o2ib5) Sep 27 17:15:19 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 17:15:19 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3772 previous similar messages Sep 27 17:15:31 oak-md1-s2 kernel: LustreError: 10495:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.0.2.247@o2ib5) returned error from blocking AST (req@ffff881880b58c00 x1610628401484128 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880e6c0c7400/0x6c00c3f3f19914ba lrc: 4/0,0 mode: PR/PR res: [0x200017967:0x2b:0x0].0x0 bits 0x13 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.0.2.247@o2ib5 remote: 0xd5af6f327dd5ac60 expref: 155485 pid: 10552 timeout: 6378023985 lvb_type: 0 Sep 27 17:15:31 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.0.2.247@o2ib5 was evicted due to a lock blocking callback time out: rc -107 Sep 27 17:15:36 oak-md1-s2 kernel: LustreError: 10495:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880008f8d100 x1610628401487456/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 27 17:15:37 oak-md1-s2 kernel: LustreError: 10612:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880f4a568300 x1610628401488544/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Sep 27 17:26:06 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 17:26:06 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5070 previous similar messages Sep 27 17:36:09 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 17:36:09 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3391 previous similar messages Sep 27 17:46:14 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 17:46:14 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3053 previous similar messages Sep 27 17:56:21 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 17:56:21 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4147 previous similar messages Sep 27 18:06:24 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 18:06:24 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4801 previous similar messages Sep 27 18:16:30 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 18:16:30 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3217 previous similar messages Sep 27 18:26:42 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 18:26:42 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3833 previous similar messages Sep 27 18:36:56 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 27 18:36:56 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4025 previous similar messages Sep 27 18:43:24 oak-md1-s2 kernel: Lustre: 10599:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538098997/real 0] req@ffff881494ed1200 x1610628404541696/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1538099004 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 27 18:43:24 oak-md1-s2 kernel: Lustre: 10599:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 27 18:44:28 oak-md1-s2 kernel: Lustre: 10652:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538099061/real 0] req@ffff880062210600 x1610628404570128/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1538099068 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Sep 27 18:44:36 oak-md1-s2 kernel: LNetError: 3778:0:(o2iblnd_cb.c:3147:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 12 seconds Sep 27 18:44:36 oak-md1-s2 kernel: LNetError: 3778:0:(o2iblnd_cb.c:3222:kiblnd_check_conns()) Timed out RDMA with 10.0.2.3@o2ib5 (62): c: 0, oc: 0, rc: 8 Sep 27 18:44:48 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.3@o2ib5: 24 seconds Sep 27 18:45:13 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.3@o2ib5: 49 seconds Sep 27 18:45:13 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 9 previous similar messages Sep 27 18:45:51 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.3@o2ib5: 0 seconds Sep 27 18:45:51 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 8 previous similar messages Sep 27 18:46:37 oak-md1-s2 kernel: LNet: Service thread pid 10599 was inactive for 200.61s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 27 18:46:37 oak-md1-s2 kernel: Pid: 10599, comm: mdt01_068 Sep 27 18:46:37 oak-md1-s2 kernel: Sep 27 18:46:37 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 27 18:46:37 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 27 18:46:37 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 27 18:46:37 oak-md1-s2 kernel: [] ? ptlrpc_interrupted_set+0x0/0x110 [ptlrpc] Sep 27 18:46:37 oak-md1-s2 kernel: [] ptlrpc_set_wait+0x4c0/0x910 [ptlrpc] Sep 27 18:46:37 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 27 18:46:37 oak-md1-s2 kernel: [] ldlm_run_ast_work+0xd3/0x3a0 [ptlrpc] Sep 27 18:46:37 oak-md1-s2 kernel: [] ldlm_handle_conflict_lock+0x75/0x330 [ptlrpc] Sep 27 18:46:37 oak-md1-s2 kernel: [] ldlm_process_inodebits_lock+0xfd/0x400 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x42e/0x970 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? ldlm_lock_create+0x1fc/0xa30 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x1c3/0x860 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] mdt_reint_object_lock+0x20/0x60 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] mdt_reint_setattr+0x60c/0x1050 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] mdt_reint+0x67/0x140 [mdt] Sep 27 18:46:38 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 27 18:46:38 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 27 18:46:38 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 27 18:46:38 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 27 18:46:38 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:46:38 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 27 18:46:38 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:46:38 oak-md1-s2 kernel: Sep 27 18:46:38 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1538099198.10599 Sep 27 18:47:07 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.3@o2ib5: 25 seconds Sep 27 18:47:07 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 37 previous similar messages Sep 27 18:47:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809409c7400, cur 1538099240 expire 1538099090 last 1538099013 Sep 27 18:47:42 oak-md1-s2 kernel: LNet: Service thread pid 10652 was inactive for 200.63s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 27 18:47:42 oak-md1-s2 kernel: Pid: 10652, comm: mdt00_102 Sep 27 18:47:42 oak-md1-s2 kernel: Sep 27 18:47:42 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 27 18:47:42 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 27 18:47:42 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 27 18:47:42 oak-md1-s2 kernel: [] ? ptlrpc_interrupted_set+0x0/0x110 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ptlrpc_set_wait+0x4c0/0x910 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 27 18:47:42 oak-md1-s2 kernel: [] ldlm_run_ast_work+0xd3/0x3a0 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ldlm_handle_conflict_lock+0x75/0x330 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ldlm_process_inodebits_lock+0xfd/0x400 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x42e/0x970 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? ldlm_lock_create+0x1fc/0xa30 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x1c3/0x860 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] mdt_reint_object_lock+0x20/0x60 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] mdt_reint_setattr+0x60c/0x1050 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] mdt_reint+0x67/0x140 [mdt] Sep 27 18:47:42 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 27 18:47:42 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 27 18:47:42 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 27 18:47:42 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 27 18:47:42 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:47:43 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 27 18:47:43 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:47:43 oak-md1-s2 kernel: Sep 27 18:47:43 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1538099263.10652 Sep 27 18:47:46 oak-md1-s2 kernel: LNet: Service thread pid 10574 was inactive for 200.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 27 18:47:46 oak-md1-s2 kernel: Pid: 10574, comm: mdt00_071 Sep 27 18:47:46 oak-md1-s2 kernel: Sep 27 18:47:46 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 27 18:47:46 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 27 18:47:46 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 27 18:47:46 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 27 18:47:46 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] mdt_getattr_name_lock+0x8a4/0x1900 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] mdt_intent_getattr+0x2b0/0x480 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Sep 27 18:47:46 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 27 18:47:46 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 27 18:47:46 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 27 18:47:46 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 27 18:47:47 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 27 18:47:47 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 27 18:47:47 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:47:47 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 27 18:47:47 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:47:47 oak-md1-s2 kernel: Sep 27 18:47:47 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1538099267.10574 Sep 27 18:49:26 oak-md1-s2 kernel: LustreError: 10574:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1538099066, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff8803d6f2ac00/0x6c00c3f40e7504fb lrc: 3/1,0 mode: --/PR res: [0x2000167df:0x13182:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10574 timeout: 0 lvb_type: 0 Sep 27 18:49:26 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.3@o2ib5: 1 seconds Sep 27 18:49:26 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 65 previous similar messages Sep 27 18:49:32 oak-md1-s2 kernel: LNet: Service thread pid 10594 was inactive for 262.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 27 18:49:32 oak-md1-s2 kernel: Pid: 10594, comm: mdt01_063 Sep 27 18:49:32 oak-md1-s2 kernel: Sep 27 18:49:32 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 27 18:49:32 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Sep 27 18:49:32 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Sep 27 18:49:32 oak-md1-s2 kernel: [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] Sep 27 18:49:32 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 27 18:49:32 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 27 18:49:33 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_object_lock+0x14/0x20 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_reint_open+0xc71/0x3190 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 27 18:49:33 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 27 18:49:33 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 27 18:49:33 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 27 18:49:33 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:49:33 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 27 18:49:33 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:49:33 oak-md1-s2 kernel: Sep 27 18:49:33 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1538099373.10594 Sep 27 18:49:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 468934c8-bd25-066c-7ba5-fc8ddaac13d0 (at 10.12.4.80@o2ib) Sep 27 18:50:10 oak-md1-s2 kernel: LustreError: 10594:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1538099110, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff8805ca42e600/0x6c00c3f40eb8e19d lrc: 3/0,1 mode: --/PW res: [0x200013a9e:0x1dd6c:0x0].0x91fef286 bits 0x2 rrc: 3 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10594 timeout: 0 lvb_type: 0 Sep 27 18:51:36 oak-md1-s2 kernel: LustreError: 10528:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1538099196, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff881865af7600/0x6c00c3f40f22db6d lrc: 3/1,0 mode: --/PR res: [0x200013a9e:0x1dd6c:0x0].0x0 bits 0x13 rrc: 18 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10528 timeout: 0 lvb_type: 0 Sep 27 18:53:12 oak-md1-s2 kernel: Lustre: 10598:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Sep 27 18:53:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 23668d2e-8d45-8686-a9df-18ea5ba33904 (at 10.0.2.251@o2ib5) reconnecting Sep 27 18:53:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 684d53b4-3df6-9d5b-0506-1d1fcbfe8402 (at 10.0.2.251@o2ib5) Sep 27 18:53:29 oak-md1-s2 kernel: LNet: Service thread pid 10528 was inactive for 412.96s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Sep 27 18:53:29 oak-md1-s2 kernel: Pid: 10528, comm: mdt01_036 Sep 27 18:53:29 oak-md1-s2 kernel: Sep 27 18:53:29 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] schedule+0x29/0x70 Sep 27 18:53:29 oak-md1-s2 kernel: [] ldlm_completion_ast+0x63d/0x920 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Sep 27 18:53:29 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? osd_fld_lookup+0x48/0xd0 [osd_ldiskfs] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:53:29 oak-md1-s2 kernel: [] mdt_object_local_lock+0x50a/0xaf0 [mdt] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? mdd_la_get+0x81/0x1a0 [mdd] Sep 27 18:53:29 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Sep 27 18:53:29 oak-md1-s2 kernel: [] mdt_getattr_name_lock+0x8a4/0x1900 [mdt] Sep 27 18:53:29 oak-md1-s2 kernel: [] mdt_intent_getattr+0x2b0/0x480 [mdt] Sep 27 18:53:29 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Sep 27 18:53:29 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Sep 27 18:53:29 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Sep 27 18:53:29 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Sep 27 18:53:30 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Sep 27 18:53:30 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Sep 27 18:53:30 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Sep 27 18:53:30 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:53:30 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Sep 27 18:53:30 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Sep 27 18:53:30 oak-md1-s2 kernel: Sep 27 18:53:30 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1538099610.10528 Sep 27 18:53:51 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.3@o2ib5: 0 seconds Sep 27 18:53:51 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 151 previous similar messages Sep 27 18:54:16 oak-md1-s2 kernel: Lustre: 10606:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Sep 27 18:54:21 oak-md1-s2 kernel: Lustre: 10513:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Sep 27 18:54:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aa81e933-816d-edb4-a86d-a0522ce2869c (at 10.12.4.82@o2ib) Sep 27 18:54:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8ac9bea1-f99e-a0f9-677c-0f3393773eb4 (at 10.8.0.67@o2ib6) reconnecting Sep 27 18:54:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f1c5637e-d3cd-5a7e-2612-b2864ebbdd0f (at 10.8.6.36@o2ib6) reconnecting Sep 27 18:54:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b15183a6-3a40-f2ab-391b-493af7b7ad4f (at 10.8.6.36@o2ib6) Sep 27 18:54:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 27 18:55:05 oak-md1-s2 kernel: Lustre: 4679:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply Sep 27 18:55:07 oak-md1-s2 kernel: LustreError: 10552:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1538099406, 300s ago); not entering recovery in server code, just going back to sleep ns: mdt-oak-MDT0000_UUID lock: ffff880013bb4c00/0x6c00c3f40fdd570b lrc: 3/1,0 mode: --/PR res: [0x200013a9e:0x1dd6c:0x0].0x0 bits 0x13 rrc: 23 type: IBT flags: 0x40210000000000 nid: local remote: 0x0 expref: -99 pid: 10552 timeout: 0 lvb_type: 0 Sep 27 18:55:10 oak-md1-s2 kernel: Lustre: 10599:0:(service.c:2114:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:113s); client may timeout. req@ffff88171abcf500 x1610806062108608/t243011178089(0) o36->23668d2e-8d45-8686-a9df-18ea5ba33904@10.0.2.251@o2ib5:7/0 lens 608/424 e 24 to 0 dl 1538099597 ref 1 fl Complete:/0/0 rc 0/0 Sep 27 18:55:10 oak-md1-s2 kernel: LNet: Service thread pid 10599 completed after 713.78s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Sep 27 18:55:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 42c9eb5b-2754-f338-5902-00d2d3c8822d (at 10.9.0.64@o2ib4) reconnecting Sep 27 18:55:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Sep 27 18:55:28 oak-md1-s2 kernel: Lustre: 10652:0:(service.c:2114:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:67s); client may timeout. req@ffff88029d8d3000 x1610691219334912/t243011221836(0) o36->8ac9bea1-f99e-a0f9-677c-0f3393773eb4@10.8.0.67@o2ib6:71/0 lens 608/424 e 24 to 0 dl 1538099661 ref 1 fl Complete:/0/0 rc 0/0 Sep 27 18:55:28 oak-md1-s2 kernel: LNet: Service thread pid 10574 completed after 662.58s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Sep 27 18:55:28 oak-md1-s2 kernel: Lustre: 10652:0:(service.c:2114:ptlrpc_server_handle_request()) Skipped 2 previous similar messages Sep 28 02:07:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fb9e235e-cf55-4e0b-2992-a0521ed9e8cb (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a4a0400, cur 1538125643 expire 1538125493 last 1538125416 Sep 28 02:32:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4966203-761f-6dc5-9911-bf9e91bbb908 (at 10.8.13.24@o2ib6) Sep 28 02:35:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 78253667-c798-cae1-cad0-4d37e581e0b3 (at 10.8.13.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880020d05c00, cur 1538127300 expire 1538127150 last 1538127073 Sep 28 07:25:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 436eb6ca-a89b-3288-4ef9-675e85a046f0 (at 10.8.1.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081fe4cc00, cur 1538144721 expire 1538144571 last 1538144494 Sep 28 09:53:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200bed2400, cur 1538153632 expire 1538153482 last 1538153405 Sep 28 11:26:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) Sep 28 12:26:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 58df6173-af81-3f2b-527c-d83552a177c3 (at 10.12.4.82@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88076f78b400, cur 1538162813 expire 1538162663 last 1538162586 Sep 28 13:20:33 oak-md1-s2 kernel: LNet: 154499:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.230@o2ib5 version 12/12 incarnation 1536014557234126/1538166031500974 Sep 28 13:20:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 589340ed-28cc-9e4a-ea13-515b211cd26f (at 10.0.2.230@o2ib5) Sep 28 13:20:53 oak-md1-s2 kernel: LNet: 154499:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.225@o2ib5 version 12/12 incarnation 1536014601731180/1538166052475237 Sep 28 13:20:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 316d4d97-f921-6627-ff1c-f0675198a07c (at 10.0.2.225@o2ib5) Sep 28 13:23:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 316d4d97-f921-6627-ff1c-f0675198a07c (at 10.0.2.225@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103d67b800, cur 1538166186 expire 1538166036 last 1538165959 Sep 28 14:13:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7387620a-60d6-5976-f752-219443e8caae (at 10.8.12.34@o2ib6) reconnecting Sep 28 14:13:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46f51a37-bd31-ab3e-e73a-5aa75fa2a817 (at 10.8.12.34@o2ib6) Sep 28 15:46:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e797d18e-1201-3e75-63d8-bdb539ab46bc (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac930c00, cur 1538174812 expire 1538174662 last 1538174585 Sep 28 15:46:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 28 16:18:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) Sep 28 16:31:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7551733b-8d16-a97a-0e6d-91c2389f4c95 (at 10.12.4.53@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be61000, cur 1538177496 expire 1538177346 last 1538177269 Sep 28 16:32:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7551733b-8d16-a97a-0e6d-91c2389f4c95 (at 10.12.4.53@o2ib) Sep 28 17:25:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 28 17:26:10 oak-md1-s2 kernel: Lustre: 10653:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:26:12 oak-md1-s2 kernel: Lustre: 10513:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:26:14 oak-md1-s2 kernel: Lustre: 10513:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:26:14 oak-md1-s2 kernel: Lustre: 10513:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 3 previous similar messages Sep 28 17:26:14 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 17:26:14 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2986 previous similar messages Sep 28 17:26:16 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:26:16 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 26 previous similar messages Sep 28 17:26:21 oak-md1-s2 kernel: Lustre: 10524:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:26:21 oak-md1-s2 kernel: Lustre: 10524:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 4 previous similar messages Sep 28 17:26:30 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:26:30 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 4 previous similar messages Sep 28 17:26:47 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:26:47 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 44 previous similar messages Sep 28 17:27:20 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:27:20 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 62 previous similar messages Sep 28 17:27:31 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 17:27:31 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1051 previous similar messages Sep 28 17:28:26 oak-md1-s2 kernel: Lustre: 10589:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:28:26 oak-md1-s2 kernel: Lustre: 10589:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 108 previous similar messages Sep 28 17:30:15 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 17:30:15 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1038 previous similar messages Sep 28 17:30:34 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:30:34 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 125 previous similar messages Sep 28 17:30:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) Sep 28 17:34:51 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:34:51 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 522 previous similar messages Sep 28 17:35:15 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 17:35:15 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2585 previous similar messages Sep 28 17:43:24 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:43:24 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 538 previous similar messages Sep 28 17:45:33 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 17:45:33 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2160 previous similar messages Sep 28 17:53:24 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 17:53:24 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 495 previous similar messages Sep 28 17:56:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9ce65db-27c6-9d52-8331-244bb4c68878 (at 10.8.1.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab0862400, cur 1538182613 expire 1538182463 last 1538182386 Sep 28 17:57:06 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 17:57:06 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2317 previous similar messages Sep 28 17:57:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7a799079-2cdd-bbe9-600e-180f51f22ca1 (at 10.8.1.31@o2ib6) Sep 28 18:03:24 oak-md1-s2 kernel: Lustre: 10589:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 18:03:24 oak-md1-s2 kernel: Lustre: 10589:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 414 previous similar messages Sep 28 18:07:19 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 18:07:19 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1832 previous similar messages Sep 28 18:12:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 266ffd14-80a7-94c1-ce9a-4b93b605506c (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801393c9400, cur 1538183532 expire 1538183382 last 1538183305 Sep 28 18:13:25 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 18:13:25 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 508 previous similar messages Sep 28 18:17:35 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 18:17:35 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1461 previous similar messages Sep 28 18:23:27 oak-md1-s2 kernel: Lustre: 10524:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 18:23:27 oak-md1-s2 kernel: Lustre: 10524:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 420 previous similar messages Sep 28 18:27:43 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 18:27:43 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1425 previous similar messages Sep 28 18:29:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 28 18:30:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3f80dae6-b53f-f49b-600d-9f98b2cb8bad (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2d632c00, cur 1538184604 expire 1538184454 last 1538184377 Sep 28 18:33:27 oak-md1-s2 kernel: Lustre: 3867:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 18:33:27 oak-md1-s2 kernel: Lustre: 3867:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 485 previous similar messages Sep 28 18:38:26 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 18:38:26 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 993 previous similar messages Sep 28 18:43:29 oak-md1-s2 kernel: Lustre: 10653:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 18:43:29 oak-md1-s2 kernel: Lustre: 10653:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 335 previous similar messages Sep 28 18:48:32 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 18:48:32 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1225 previous similar messages Sep 28 18:53:31 oak-md1-s2 kernel: Lustre: 10574:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 18:53:31 oak-md1-s2 kernel: Lustre: 10574:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 343 previous similar messages Sep 28 18:58:37 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 18:58:37 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1420 previous similar messages Sep 28 19:03:34 oak-md1-s2 kernel: Lustre: 10676:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 19:03:34 oak-md1-s2 kernel: Lustre: 10676:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 515 previous similar messages Sep 28 19:08:42 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 19:08:42 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1410 previous similar messages Sep 28 19:13:37 oak-md1-s2 kernel: Lustre: 10651:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 19:13:37 oak-md1-s2 kernel: Lustre: 10651:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 355 previous similar messages Sep 28 19:19:01 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 19:19:01 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1467 previous similar messages Sep 28 19:23:39 oak-md1-s2 kernel: Lustre: 10588:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 19:23:39 oak-md1-s2 kernel: Lustre: 10588:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 352 previous similar messages Sep 28 19:31:45 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 19:31:45 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2019 previous similar messages Sep 28 19:33:42 oak-md1-s2 kernel: Lustre: 10495:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 19:33:42 oak-md1-s2 kernel: Lustre: 10495:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 365 previous similar messages Sep 28 19:42:31 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 19:42:31 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 999 previous similar messages Sep 28 19:43:45 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 19:43:45 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 426 previous similar messages Sep 28 19:53:48 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 19:53:48 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 308 previous similar messages Sep 28 20:03:49 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 20:03:49 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 973 previous similar messages Sep 28 20:13:55 oak-md1-s2 kernel: Lustre: 10485:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 20:13:55 oak-md1-s2 kernel: Lustre: 10485:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 90 previous similar messages Sep 28 20:23:59 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 20:23:59 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 74 previous similar messages Sep 28 20:34:02 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 20:34:02 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 75 previous similar messages Sep 28 20:44:03 oak-md1-s2 kernel: Lustre: 10676:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 20:44:03 oak-md1-s2 kernel: Lustre: 10676:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 67 previous similar messages Sep 28 20:54:05 oak-md1-s2 kernel: Lustre: 10639:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 20:54:05 oak-md1-s2 kernel: Lustre: 10639:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 71 previous similar messages Sep 28 20:56:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ea48c13-0324-666d-ccd1-9ab1a795e5f9 (at 10.8.18.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f9775a800, cur 1538193372 expire 1538193222 last 1538193145 Sep 28 21:04:10 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 21:04:10 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 94 previous similar messages Sep 28 21:14:12 oak-md1-s2 kernel: Lustre: 4303:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 21:14:12 oak-md1-s2 kernel: Lustre: 4303:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 128 previous similar messages Sep 28 21:24:15 oak-md1-s2 kernel: Lustre: 10653:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 21:24:15 oak-md1-s2 kernel: Lustre: 10653:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 159 previous similar messages Sep 28 21:34:15 oak-md1-s2 kernel: Lustre: 10545:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 21:34:15 oak-md1-s2 kernel: Lustre: 10545:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1081 previous similar messages Sep 28 21:44:15 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 21:44:15 oak-md1-s2 kernel: Lustre: 10606:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1571 previous similar messages Sep 28 21:45:51 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 21:45:51 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1617 previous similar messages Sep 28 21:54:16 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 21:54:16 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1419 previous similar messages Sep 28 22:04:16 oak-md1-s2 kernel: Lustre: 10589:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 22:04:16 oak-md1-s2 kernel: Lustre: 10589:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 863 previous similar messages Sep 28 22:14:19 oak-md1-s2 kernel: Lustre: 4256:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 22:14:19 oak-md1-s2 kernel: Lustre: 4256:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 353 previous similar messages Sep 28 22:23:45 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 22:23:45 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Sep 28 22:24:20 oak-md1-s2 kernel: Lustre: 10588:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 22:24:20 oak-md1-s2 kernel: Lustre: 10588:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 252 previous similar messages Sep 28 22:34:22 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 22:34:22 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 276 previous similar messages Sep 28 22:44:23 oak-md1-s2 kernel: Lustre: 10655:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 22:44:23 oak-md1-s2 kernel: Lustre: 10655:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 183 previous similar messages Sep 28 22:54:24 oak-md1-s2 kernel: Lustre: 10513:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 22:54:24 oak-md1-s2 kernel: Lustre: 10513:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 471 previous similar messages Sep 28 23:04:25 oak-md1-s2 kernel: Lustre: 10495:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 23:04:25 oak-md1-s2 kernel: Lustre: 10495:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 161 previous similar messages Sep 28 23:05:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0fbfb7a2-9c91-f08e-b8db-ff8d2b652a0f (at 10.12.4.59@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880de3af0800, cur 1538201115 expire 1538200965 last 1538200888 Sep 28 23:11:46 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 23:14:23 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 23:14:23 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Sep 28 23:14:25 oak-md1-s2 kernel: Lustre: 10612:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 23:14:25 oak-md1-s2 kernel: Lustre: 10612:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 158 previous similar messages Sep 28 23:24:28 oak-md1-s2 kernel: Lustre: 10630:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 23:24:28 oak-md1-s2 kernel: Lustre: 10630:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 137 previous similar messages Sep 28 23:29:38 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 28 23:29:38 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 28 previous similar messages Sep 28 23:34:32 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 23:34:32 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 102 previous similar messages Sep 28 23:44:38 oak-md1-s2 kernel: Lustre: 10576:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 23:44:38 oak-md1-s2 kernel: Lustre: 10576:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 134 previous similar messages Sep 28 23:54:39 oak-md1-s2 kernel: Lustre: 3867:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 28 23:54:39 oak-md1-s2 kernel: Lustre: 3867:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 149 previous similar messages Sep 29 00:04:40 oak-md1-s2 kernel: Lustre: 4694:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 00:04:40 oak-md1-s2 kernel: Lustre: 4694:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 243 previous similar messages Sep 29 00:14:43 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 00:14:43 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 227 previous similar messages Sep 29 00:24:49 oak-md1-s2 kernel: Lustre: 3867:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 00:24:49 oak-md1-s2 kernel: Lustre: 3867:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 246 previous similar messages Sep 29 00:34:50 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 00:34:50 oak-md1-s2 kernel: Lustre: 10656:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 189 previous similar messages Sep 29 00:44:02 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 00:44:02 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 29 00:44:50 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 00:44:50 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 208 previous similar messages Sep 29 00:54:51 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 00:54:51 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 214 previous similar messages Sep 29 01:04:53 oak-md1-s2 kernel: Lustre: 10524:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 01:04:53 oak-md1-s2 kernel: Lustre: 10524:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 169 previous similar messages Sep 29 01:14:54 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 01:14:54 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 153 previous similar messages Sep 29 01:24:57 oak-md1-s2 kernel: Lustre: 10545:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 01:24:57 oak-md1-s2 kernel: Lustre: 10545:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 139 previous similar messages Sep 29 01:32:57 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 01:32:57 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 29 01:35:01 oak-md1-s2 kernel: Lustre: 10556:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 01:35:01 oak-md1-s2 kernel: Lustre: 10556:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 162 previous similar messages Sep 29 01:40:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 01:41:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db91b824-a518-4c60-1179-0e2b73444cb7 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a5147c000, cur 1538210474 expire 1538210324 last 1538210247 Sep 29 01:45:01 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 01:45:01 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages Sep 29 01:45:05 oak-md1-s2 kernel: Lustre: 10515:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 01:45:05 oak-md1-s2 kernel: Lustre: 10515:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 159 previous similar messages Sep 29 01:48:56 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 01:48:56 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Sep 29 01:51:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 46b8e74c-d69e-ca93-db21-a4fd73a05c2b (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aeff00800, cur 1538211064 expire 1538210914 last 1538210837 Sep 29 01:55:06 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 01:55:06 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 151 previous similar messages Sep 29 01:59:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 02:05:07 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 02:05:07 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 154 previous similar messages Sep 29 02:13:10 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 02:13:10 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Sep 29 02:15:08 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 02:15:08 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 208 previous similar messages Sep 29 02:19:04 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 02:19:04 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Sep 29 02:21:49 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 02:21:49 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 34 previous similar messages Sep 29 02:23:59 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 02:23:59 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Sep 29 02:25:10 oak-md1-s2 kernel: Lustre: 10544:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 02:25:10 oak-md1-s2 kernel: Lustre: 10544:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 256 previous similar messages Sep 29 02:28:03 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 02:28:03 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 55 previous similar messages Sep 29 02:28:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 987663f4-eeb6-336b-3d4d-a47f712d600a (at 10.8.6.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880353c3ac00, cur 1538213317 expire 1538213167 last 1538213090 Sep 29 02:35:12 oak-md1-s2 kernel: Lustre: 10544:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 02:35:12 oak-md1-s2 kernel: Lustre: 10544:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 270 previous similar messages Sep 29 02:35:38 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 02:35:38 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 39 previous similar messages Sep 29 02:39:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1c92d11a-e2f1-f513-c7f8-2b6130047946 (at 10.8.6.14@o2ib6) Sep 29 02:39:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5a524ba4-15c7-1347-2921-cd73173ea987 (at 10.12.4.33@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be03000, cur 1538213951 expire 1538213801 last 1538213724 Sep 29 02:40:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a524ba4-15c7-1347-2921-cd73173ea987 (at 10.12.4.33@o2ib) Sep 29 02:45:14 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 02:45:14 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 233 previous similar messages Sep 29 02:53:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7539470b-de0c-5e55-cd27-3c93967dfb9d (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edf6400, cur 1538214793 expire 1538214643 last 1538214566 Sep 29 02:53:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 02:55:15 oak-md1-s2 kernel: Lustre: 10544:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 02:55:15 oak-md1-s2 kernel: Lustre: 10544:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 197 previous similar messages Sep 29 02:58:16 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 02:58:16 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Sep 29 03:02:31 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 03:02:31 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Sep 29 03:02:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 03:03:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9878321a-e5e1-88c7-0a63-6fe11003a9ac (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802267e2800, cur 1538215411 expire 1538215261 last 1538215184 Sep 29 03:05:16 oak-md1-s2 kernel: Lustre: 4303:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 03:05:16 oak-md1-s2 kernel: Lustre: 4303:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 177 previous similar messages Sep 29 03:15:19 oak-md1-s2 kernel: Lustre: 10605:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 03:15:19 oak-md1-s2 kernel: Lustre: 10605:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 155 previous similar messages Sep 29 03:25:21 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 03:25:21 oak-md1-s2 kernel: Lustre: 10550:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 155 previous similar messages Sep 29 03:29:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client acdd75fe-b325-54ad-3e7e-2cce9f7641e3 (at 10.12.4.83@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200bed7c00, cur 1538216948 expire 1538216798 last 1538216721 Sep 29 03:35:27 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 03:35:27 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 135 previous similar messages Sep 29 03:42:19 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 03:42:19 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Sep 29 03:45:28 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 03:45:28 oak-md1-s2 kernel: Lustre: 10504:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 125 previous similar messages Sep 29 03:49:31 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 03:49:31 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages Sep 29 03:49:32 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 03:55:30 oak-md1-s2 kernel: Lustre: 10492:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 03:55:30 oak-md1-s2 kernel: Lustre: 10492:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 123 previous similar messages Sep 29 04:00:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 04:00:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99fc2c58-22b6-0368-9be9-801a0dbc3c54 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7c630000, cur 1538218849 expire 1538218699 last 1538218622 Sep 29 04:05:35 oak-md1-s2 kernel: Lustre: 10630:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 04:05:35 oak-md1-s2 kernel: Lustre: 10630:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 150 previous similar messages Sep 29 04:07:32 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:07:32 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Sep 29 04:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 04:10:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b765bd26-6bb8-4d2d-a965-9372bab4f920 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880af70d3c00, cur 1538219408 expire 1538219258 last 1538219181 Sep 29 04:15:37 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 04:15:37 oak-md1-s2 kernel: Lustre: 10652:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 184 previous similar messages Sep 29 04:19:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 04:20:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e794513e-412c-268d-e324-5e7e26d649a4 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc3a800, cur 1538220044 expire 1538219894 last 1538219817 Sep 29 04:25:02 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:25:02 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 44 previous similar messages Sep 29 04:25:41 oak-md1-s2 kernel: Lustre: 10558:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 04:25:41 oak-md1-s2 kernel: Lustre: 10558:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 225 previous similar messages Sep 29 04:35:44 oak-md1-s2 kernel: Lustre: 10495:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 04:35:44 oak-md1-s2 kernel: Lustre: 10495:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 181 previous similar messages Sep 29 04:36:40 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:36:40 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Sep 29 04:36:41 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:36:41 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 37 previous similar messages Sep 29 04:40:55 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:40:55 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 29 previous similar messages Sep 29 04:45:48 oak-md1-s2 kernel: Lustre: 10556:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 04:45:48 oak-md1-s2 kernel: Lustre: 10556:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 179 previous similar messages Sep 29 04:55:10 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:55:10 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 58 previous similar messages Sep 29 04:55:11 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:55:11 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 37 previous similar messages Sep 29 04:55:50 oak-md1-s2 kernel: Lustre: 10676:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 04:55:50 oak-md1-s2 kernel: Lustre: 10676:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 222 previous similar messages Sep 29 04:59:44 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 04:59:44 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 29 05:05:51 oak-md1-s2 kernel: Lustre: 10655:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 05:05:51 oak-md1-s2 kernel: Lustre: 10655:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 227 previous similar messages Sep 29 05:15:54 oak-md1-s2 kernel: Lustre: 10485:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 05:15:54 oak-md1-s2 kernel: Lustre: 10485:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 228 previous similar messages Sep 29 05:22:03 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 05:22:03 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 42 previous similar messages Sep 29 05:22:05 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 05:22:05 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 50 previous similar messages Sep 29 05:22:12 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 05:22:12 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages Sep 29 05:25:55 oak-md1-s2 kernel: Lustre: 10515:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 05:25:55 oak-md1-s2 kernel: Lustre: 10515:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 190 previous similar messages Sep 29 05:35:55 oak-md1-s2 kernel: Lustre: 10651:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 05:35:55 oak-md1-s2 kernel: Lustre: 10651:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 152 previous similar messages Sep 29 05:43:42 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 05:43:42 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Sep 29 05:45:57 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 05:45:57 oak-md1-s2 kernel: Lustre: 10634:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 208 previous similar messages Sep 29 05:55:57 oak-md1-s2 kernel: Lustre: 10653:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 05:55:57 oak-md1-s2 kernel: Lustre: 10653:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 199 previous similar messages Sep 29 06:05:58 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 06:05:58 oak-md1-s2 kernel: Lustre: 10549:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 186 previous similar messages Sep 29 06:14:41 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 06:14:41 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 57 previous similar messages Sep 29 06:15:18 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 06:15:18 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 34 previous similar messages Sep 29 06:15:59 oak-md1-s2 kernel: Lustre: 10513:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Sep 29 06:15:59 oak-md1-s2 kernel: Lustre: 10513:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 160 previous similar messages Sep 29 06:17:09 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 06:17:09 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 39 previous similar messages Sep 29 06:18:38 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 06:18:38 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Sep 29 06:58:45 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 06:58:45 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 57 previous similar messages Sep 29 07:37:37 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 07:37:37 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 42 previous similar messages Sep 29 07:37:38 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 07:37:38 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 28 previous similar messages Sep 29 08:11:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d799bbe0-7924-337a-51ad-b19a59d5343f (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b643b400, cur 1538233900 expire 1538233750 last 1538233673 Sep 29 08:33:23 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 08:33:23 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 28 previous similar messages Sep 29 08:33:23 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 08:33:23 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 42 previous similar messages Sep 29 08:36:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 29 08:37:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae0ea45c-acfb-c4e3-af7b-5ed8c7978756 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e42849c00, cur 1538235438 expire 1538235288 last 1538235211 Sep 29 08:50:16 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 08:50:16 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Sep 29 09:02:23 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 09:02:23 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 39 previous similar messages Sep 29 09:02:24 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 09:02:24 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Sep 29 09:03:58 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 09:03:58 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 44 previous similar messages Sep 29 09:08:55 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 09:08:55 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 46 previous similar messages Sep 29 09:20:49 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 09:20:49 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 29 previous similar messages Sep 29 09:30:09 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 09:30:09 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 32 previous similar messages Sep 29 10:13:13 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 10:13:13 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Sep 29 11:04:33 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 11:45:26 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 11:45:26 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 11:45:26 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Sep 29 11:48:12 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 11:48:12 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Sep 29 11:50:27 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 11:50:27 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Sep 29 12:51:37 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 12:51:37 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages Sep 29 12:55:01 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 13:09:59 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 13:14:47 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 13:14:47 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Sep 29 13:30:14 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 13:30:14 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages Sep 29 13:30:14 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 13:30:14 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Sep 29 13:45:56 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 13:45:56 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Sep 29 14:52:45 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 15:09:33 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 15:09:33 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Sep 29 19:16:41 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 19:16:41 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Sep 29 21:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aa81e933-816d-edb4-a86d-a0522ce2869c (at 10.12.4.82@o2ib) Sep 29 21:53:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acdd75fe-b325-54ad-3e7e-2cce9f7641e3 (at 10.12.4.83@o2ib) Sep 29 22:32:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb7d4db9-8df8-5902-0a6f-ab6762b111b8 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c0ea44800, cur 1538285525 expire 1538285375 last 1538285298 Sep 29 22:46:12 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 29 22:46:12 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 29 previous similar messages Sep 29 23:13:20 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Sep 30 04:01:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02f251e8-bb9a-4758-8ec6-7b12601fe2e2 (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c648800, cur 1538305319 expire 1538305169 last 1538305092 Sep 30 05:35:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 135b16fa-86e7-b216-1670-5ae846d8b66e (at 10.8.2.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040e0a800, cur 1538310932 expire 1538310782 last 1538310705 Sep 30 07:45:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42ddaac4-4249-5052-00c0-3d0a27940cdb (at 10.0.2.224@o2ib5) Sep 30 07:46:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0544db67-8363-8e08-d2d9-ead73d3dc54c (at 10.0.2.224@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daca08400, cur 1538318819 expire 1538318669 last 1538318592 Sep 30 07:49:19 oak-md1-s2 kernel: LNet: 154499:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.224@o2ib5 version 12/12 incarnation 1538318748960533/1538318958121204 Sep 30 07:49:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42ddaac4-4249-5052-00c0-3d0a27940cdb (at 10.0.2.224@o2ib5) Sep 30 08:57:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client faf6b65d-2d95-59fd-3105-aecf2acabfe1 (at 10.12.4.83@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe1347800, cur 1538323060 expire 1538322910 last 1538322833 Sep 30 09:01:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acdd75fe-b325-54ad-3e7e-2cce9f7641e3 (at 10.12.4.83@o2ib) Sep 30 12:19:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 24285770-01af-0f32-d97e-c5b23a9d6b85 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040e0c800, cur 1538335172 expire 1538335022 last 1538334945 Sep 30 12:28:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 12:48:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a1cea396-4728-3ce5-c4a6-7a3cd9797074 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d39b48400, cur 1538336926 expire 1538336776 last 1538336699 Sep 30 12:57:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 13:06:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 499c7c04-c7c4-6aea-8539-6252c26a261b (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880010d66800, cur 1538338012 expire 1538337862 last 1538337785 Sep 30 13:06:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 13:15:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 13:16:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f76979c-9130-0990-0631-f347505173b3 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805bd942400, cur 1538338571 expire 1538338421 last 1538338344 Sep 30 13:24:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 13:25:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fbbbdd90-dd2a-194e-6752-1c5f9b275be9 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881569fd9400, cur 1538339119 expire 1538338969 last 1538338892 Sep 30 13:38:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 13:38:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea0130cb-1999-8dad-b30a-ad2317e56577 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88906400, cur 1538339930 expire 1538339780 last 1538339703 Sep 30 14:21:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7a8c3fc9-e82f-6ab8-3f07-62b0f3700c4d (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004f66c000, cur 1538342491 expire 1538342341 last 1538342264 Sep 30 14:30:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 17:55:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2b32d60e-17c0-3d0b-c4ed-75dd9ae1d0f6 (at 10.8.18.4@o2ib6) Sep 30 17:55:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34263530-dddb-d9e6-8e5d-2c189bd873b2 (at 10.8.27.33@o2ib6) Sep 30 17:55:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Sep 30 17:55:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4cd4f04f-3cdc-6f4c-6397-b6cc485356f7 (at 10.8.17.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9441000, cur 1538355341 expire 1538355191 last 1538355114 Sep 30 17:57:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d85dd6b-c92b-660d-fcaa-6da2163a1f77 (at 10.8.1.14@o2ib6) Sep 30 17:57:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b9a3194e-19c3-f27c-34cc-b6dd41be449b (at 10.8.7.19@o2ib6) Sep 30 18:04:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c28e98af-9208-6806-919b-07aacb13fb09 (at 10.8.27.9@o2ib6) Sep 30 18:11:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 14599db0-f386-f663-8087-f701473ab91f (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e3875b000, cur 1538356317 expire 1538356167 last 1538356090 Sep 30 18:11:57 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Sep 30 18:33:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3dbf491e-bdc6-b54e-96ef-870125d582a0 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807fa9abc00, cur 1538357630 expire 1538357480 last 1538357403 Sep 30 18:34:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 19:19:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 93746428-2f58-7d78-871d-b7b2cb66dc27 (at 10.8.18.20@o2ib6) Sep 30 19:20:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 10333f1a-8454-9045-d2e4-c3a70fc5176d (at 10.8.18.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880004434400, cur 1538360422 expire 1538360272 last 1538360195 Sep 30 19:33:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a91899a1-5697-b42c-9a7c-544ebef08626 (at 10.8.13.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808d6af7c00, cur 1538361193 expire 1538361043 last 1538360966 Sep 30 19:49:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59785839-88e4-8789-c5bd-4379aa18fb93 (at 10.8.18.17@o2ib6) Sep 30 19:50:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c8ffbb94-fc75-3c5c-2999-58d193930074 (at 10.8.18.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809331be800, cur 1538362248 expire 1538362098 last 1538362021 Sep 30 20:27:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Sep 30 20:27:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 140f306e-b107-3f15-0e78-e1149aeab544 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e4000, cur 1538364464 expire 1538364314 last 1538364237 Sep 30 20:46:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7dffbb55-f764-db73-b15e-47931eae5324 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d1800, cur 1538365593 expire 1538365443 last 1538365366 Sep 30 20:46:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 21:21:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0bcbc27e-a2ea-ec0f-4344-c2297f0c44ae (at 10.8.17.13@o2ib6) Sep 30 21:22:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f585212-1eb4-c9a0-36ab-0b36271c9fcb (at 10.8.17.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880675bfdc00, cur 1538367770 expire 1538367620 last 1538367543 Sep 30 21:23:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f4a61592-7d40-9916-52cf-6afa19fcae00 (at 10.8.1.22@o2ib6) Sep 30 21:29:00 oak-md1-s2 kernel: LNet: 3782:0:(o2iblnd_cb.c:396:kiblnd_handle_rx()) PUT_NACK from 10.0.2.3@o2ib5 Sep 30 21:29:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Sep 30 21:29:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 30 21:29:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Sep 30 21:29:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Sep 30 21:33:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4cd85656-cf72-67a5-e479-6a1921e887c6 (at 10.8.1.1@o2ib6) Sep 30 21:37:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f1c92641-c7dc-ee81-2dbd-aebfc60a2d86 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe9df3c00, cur 1538368624 expire 1538368474 last 1538368397 Sep 30 21:37:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Sep 30 21:37:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 21:46:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 21:46:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52c8d370-fcde-84f5-f1f2-5e493c4333fe (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005a0cd400, cur 1538369203 expire 1538369053 last 1538368976 Sep 30 21:55:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f855068-9c80-123d-cd9c-faf018b9ee39 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880541150400, cur 1538369737 expire 1538369587 last 1538369510 Sep 30 21:55:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 22:03:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8986f6af-ef08-91b2-3d4d-7b0a6bf9090a (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88070479a000, cur 1538370204 expire 1538370054 last 1538369977 Sep 30 22:04:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 22:14:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a59ec1d3-c54a-a0e7-6d87-bc7bfc26d41d (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5285f000, cur 1538370889 expire 1538370739 last 1538370662 Sep 30 22:16:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 22:30:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aaf9635d-b865-ab41-92ee-43b984eae489 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe1347c00, cur 1538371815 expire 1538371665 last 1538371588 Sep 30 22:30:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 22:37:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 22:38:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec885635-ccc6-18c7-03c6-7888b47a040b (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a5147e000, cur 1538372310 expire 1538372160 last 1538372083 Sep 30 22:39:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cac1feff-9abc-48ce-3bbb-b61c60d0d3e0 (at 10.8.17.27@o2ib6) in 205 seconds. I think it's dead, and I am evicting it. exp ffff880020d01c00, cur 1538372386 expire 1538372236 last 1538372181 Sep 30 22:41:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Sep 30 22:42:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ed030f1-cef1-7496-4337-7eed77c0b4c9 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807d4eedc00, cur 1538372551 expire 1538372401 last 1538372324 Sep 30 22:46:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 22:47:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d65ed00b-d050-3233-ec6c-c9d07b6a4ebb (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880456375000, cur 1538372821 expire 1538372671 last 1538372594 Sep 30 23:11:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Sep 30 23:12:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f127572-b473-1f0a-e9d5-8309926ece92 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e7e7f800, cur 1538374330 expire 1538374180 last 1538374103 Sep 30 23:29:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0a93a911-4fb9-9442-a2e0-f9f362357d92 (at 10.8.18.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808c62fb000, cur 1538375380 expire 1538375230 last 1538375153 Sep 30 23:49:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9feef4d8-a8f6-db4b-3c35-958b25627eea (at 10.8.1.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e1ff16800, cur 1538376567 expire 1538376417 last 1538376340 Sep 30 23:58:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Sep 30 23:58:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 31774e71-ecd3-0087-7a85-981d0946bfe9 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802267e4000, cur 1538377133 expire 1538376983 last 1538376906 Oct 01 01:34:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client abfca770-00e3-1dde-d6ff-917161d39e4d (at 10.12.4.84@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be04400, cur 1538382852 expire 1538382702 last 1538382625 Oct 01 02:56:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 01 02:56:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) Oct 01 04:02:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 01 04:03:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a7314ab9-f009-1427-883f-a625a9a6c45e (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800496b6c00, cur 1538391785 expire 1538391635 last 1538391558 Oct 01 06:00:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 89d2a802-1c89-0581-9675-ae48e97b33f4 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004f669c00, cur 1538398833 expire 1538398683 last 1538398606 Oct 01 06:09:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 01 08:00:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b82e363f-2682-19f8-c12e-347146acdfda (at 10.8.27.4@o2ib6) Oct 01 08:02:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0d6c4a89-91a4-c36f-e9db-dc452ee5c073 (at 10.8.27.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f11d800, cur 1538406127 expire 1538405977 last 1538405900 Oct 01 08:16:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3d2044e-c773-9435-a646-8ec1edb9ab66 (at 10.8.11.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b8b430400, cur 1538406989 expire 1538406839 last 1538406762 Oct 01 08:16:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 01 08:19:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5cb5b2ef-3049-d802-fb16-4de61afdcd62 (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7de6fc00, cur 1538407170 expire 1538407020 last 1538406943 Oct 01 08:21:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 01 08:21:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Oct 01 08:24:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 01 08:24:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 330bb49a-e826-005b-10d9-5c0f423f4bdf (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85a88800, cur 1538407498 expire 1538407348 last 1538407271 Oct 01 08:26:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc8bab17-3a1e-a54e-8adf-eca1a8e4aa7b (at 10.8.15.10@o2ib6) in 203 seconds. I think it's dead, and I am evicting it. exp ffff8809439dd800, cur 1538407574 expire 1538407424 last 1538407371 Oct 01 08:27:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bced131c-6b24-fe29-db28-e98ba8348efb (at 10.8.0.82@o2ib6) in 172 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afb800, cur 1538407650 expire 1538407500 last 1538407478 Oct 01 08:28:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a7a2537-1bf1-6230-67f7-9e3f322cde31 (at 10.8.3.29@o2ib6) in 200 seconds. I think it's dead, and I am evicting it. exp ffff880f19686000, cur 1538407726 expire 1538407576 last 1538407526 Oct 01 08:36:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bced131c-6b24-fe29-db28-e98ba8348efb (at 10.8.0.82@o2ib6) Oct 01 08:46:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df035d19-78a0-503f-9131-c8189af6c781 (at 10.8.0.82@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc1542c00, cur 1538408771 expire 1538408621 last 1538408544 Oct 01 08:46:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 01 08:47:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c144164-ef50-a9fe-5bea-54eb9532eceb (at 10.8.11.10@o2ib6) in 189 seconds. I think it's dead, and I am evicting it. exp ffff880751704000, cur 1538408847 expire 1538408697 last 1538408658 Oct 01 09:07:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 01 09:24:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 01 09:28:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8fe86858-0c33-94d5-0210-961bc84324b9 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88068aa28800, cur 1538411331 expire 1538411181 last 1538411104 Oct 01 09:31:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 01 09:32:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c15f9af-25c1-dd63-ea15-3f4aecfecabf (at 10.8.18.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe258e800, cur 1538411576 expire 1538411426 last 1538411349 Oct 01 09:38:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a6121e0-49d3-8d5d-d708-98776bf38e86 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d39b4c800, cur 1538411901 expire 1538411751 last 1538411674 Oct 01 09:38:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 01 09:41:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bc4545df-0f85-4787-eae8-4f37b6959728 (at 10.8.18.15@o2ib6) Oct 01 09:45:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 831aeb03-048f-5498-e47f-586861732487 (at 10.8.18.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e151eac00, cur 1538412303 expire 1538412153 last 1538412076 Oct 01 09:54:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ad32a28-c1bb-d588-df17-32e18e07516c (at 10.8.18.12@o2ib6) Oct 01 09:58:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aa675f90-c2fe-0ab6-3178-fab3e374d618 (at 10.9.0.1@o2ib4) Oct 01 09:59:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 01 10:03:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 01 10:03:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ece08209-1105-3335-e146-2f99aaea151d (at 10.8.18.21@o2ib6) Oct 01 10:08:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 01 10:10:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d13af22c-82c5-85f9-1bc5-ba124040f4ab (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb1958400, cur 1538413829 expire 1538413679 last 1538413602 Oct 01 10:11:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 01 11:15:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7eeef807-cfe6-bdf6-2869-c2a61c4c66db (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a7d333000, cur 1538417725 expire 1538417575 last 1538417498 Oct 01 11:22:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 01 11:28:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9415667a-3a90-71d7-f13c-4c0ec43bc02b (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa842800, cur 1538418520 expire 1538418370 last 1538418293 Oct 01 11:36:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 01 11:39:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0822afea-569d-1eb6-8f64-57449413c992 (at 10.12.4.47@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102fff6000, cur 1538419143 expire 1538418993 last 1538418916 Oct 01 11:53:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b153cca9-3865-0883-8336-ed057b6dc40f (at 10.9.0.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88048a9ea800, cur 1538419994 expire 1538419844 last 1538419767 Oct 01 12:15:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to abfca770-00e3-1dde-d6ff-917161d39e4d (at 10.12.4.84@o2ib) Oct 01 12:16:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c1c5ea59-40db-8ad9-fb5f-2c9fd5cf9542 (at 10.12.4.59@o2ib) Oct 01 12:20:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bced131c-6b24-fe29-db28-e98ba8348efb (at 10.8.0.82@o2ib6) Oct 01 14:09:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8a2d8d37-c52b-cb75-8d8e-4892d20c58f0 (at 10.12.4.83@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f710800, cur 1538428188 expire 1538428038 last 1538427961 Oct 01 14:09:48 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 01 14:12:08 oak-md1-s2 kernel: Lustre: DEBUG MARKER: Mon Oct 1 14:12:08 2018 Oct 01 14:24:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acdd75fe-b325-54ad-3e7e-2cce9f7641e3 (at 10.12.4.83@o2ib) Oct 01 14:28:26 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 14:28:26 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 01 14:29:52 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 14:29:52 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 01 14:34:54 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 14:34:54 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 01 14:47:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3eac4d27-6938-5bd3-6cf0-77bbdf595f11 (at 10.9.0.61@o2ib4) Oct 01 14:56:10 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 14:56:10 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 01 15:07:52 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:07:52 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 01 15:11:00 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:11:00 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 01 15:15:25 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:15:25 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 01 15:22:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 32ebea3e-2a52-fcdb-13f7-508d1b805764 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fec3d5400, cur 1538432574 expire 1538432424 last 1538432347 Oct 01 15:25:05 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:25:05 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 37 previous similar messages Oct 01 15:25:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 15:25:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 01 15:27:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 01 15:28:30 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:28:30 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 01 15:28:48 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:28:48 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 01 15:29:37 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:29:37 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 01 15:33:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3eac4d27-6938-5bd3-6cf0-77bbdf595f11 (at 10.9.0.61@o2ib4) Oct 01 15:37:19 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:37:19 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 01 15:45:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 01 15:46:10 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 15:46:10 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 01 15:54:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 15:54:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 01 15:57:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b69140d-a83b-c287-c38c-8a22abebae36 (at 10.9.0.62@o2ib4) Oct 01 16:01:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Oct 01 16:10:15 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 16:10:15 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 01 16:22:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 16:22:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 01 16:35:18 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 16:35:18 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 01 16:47:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb15d3ef-3a25-3d98-4fb5-5f36639ab2bb (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004f669000, cur 1538437678 expire 1538437528 last 1538437451 Oct 01 16:53:48 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 16:53:48 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 01 16:56:17 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 16:56:17 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 01 16:59:13 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 16:59:13 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 01 17:01:12 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:01:59 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:02:46 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:04:28 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:05:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47cf49c5-0515-7f67-da5c-d23e377ac991 (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880463a30800, cur 1538438704 expire 1538438554 last 1538438477 Oct 01 17:08:50 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:08:50 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17 previous similar messages Oct 01 17:10:53 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:11:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 17:11:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 01 17:14:04 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:14:04 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 01 17:19:45 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:19:45 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 01 17:22:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 01 17:38:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 17:38:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 01 17:38:07 oak-md1-s2 kernel: LustreError: 7156:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881d1a7e7b00 x1610627738044688/t0(0) o37->0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4@10.0.2.221@o2ib5:606/0 lens 568/440 e 0 to 0 dl 1538440701 ref 1 fl Interpret:/0/0 rc 0/0 Oct 01 17:38:27 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:38:27 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14 previous similar messages Oct 01 17:42:29 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:42:29 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 01 17:48:40 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 17:48:40 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 01 17:48:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d151c9eb-af1a-7ce6-d43e-6b743645112c (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da1082400, cur 1538441331 expire 1538441181 last 1538441104 Oct 01 17:49:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 01 17:51:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 01 17:52:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b516e166-969f-4a01-774d-d27da4f27946 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039bd93000, cur 1538441521 expire 1538441371 last 1538441294 Oct 01 17:52:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 01 17:52:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Oct 01 17:53:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d0ee3631-cc0a-82c2-9662-fa0bce6fb567 (at 10.8.27.12@o2ib6) in 212 seconds. I think it's dead, and I am evicting it. exp ffff881e415b3000, cur 1538441597 expire 1538441447 last 1538441385 Oct 01 17:53:17 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 01 17:57:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3eac4d27-6938-5bd3-6cf0-77bbdf595f11 (at 10.9.0.61@o2ib4) Oct 01 17:57:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ece17335-761a-9777-384a-8dda740880f9 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b268b800, cur 1538441871 expire 1538441721 last 1538441644 Oct 01 17:58:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 01 18:00:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30fe12c7-28d3-18c6-0aaf-ec2372c62091 (at 10.8.27.1@o2ib6) Oct 01 18:00:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3675ccf5-fdbf-12b1-81ce-36273790ee58 (at 10.8.18.30@o2ib6) Oct 01 18:02:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 01 18:02:27 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 18:02:27 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 01 18:05:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b73097e-2b86-6ca0-b2c5-9f454d65d255 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c19bc00, cur 1538442317 expire 1538442167 last 1538442090 Oct 01 18:06:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 01 18:08:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 01 18:08:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 01 18:09:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c84047e2-04d1-f93d-f094-06c523f1b26b (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851230000, cur 1538442554 expire 1538442404 last 1538442327 Oct 01 18:10:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 18:11:56 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 18:11:56 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 01 18:13:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4966203-761f-6dc5-9911-bf9e91bbb908 (at 10.8.13.24@o2ib6) Oct 01 18:13:51 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 01 18:16:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2bfcba77-5f3d-117c-730a-35713f6f2b15 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88160cee3400, cur 1538442993 expire 1538442843 last 1538442766 Oct 01 18:17:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9492a932-1abf-76ac-8309-1ab3a6508520 (at 10.8.1.7@o2ib6) in 177 seconds. I think it's dead, and I am evicting it. exp ffff880a5147b400, cur 1538443069 expire 1538442919 last 1538442892 Oct 01 18:22:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5158eccf-14ae-5d94-c749-4edeb98a44a2 (at 10.8.6.8@o2ib6) Oct 01 18:22:40 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 01 18:24:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 66ad91e7-8f8f-dab4-fa11-f2861b739794 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88070479e800, cur 1538443466 expire 1538443316 last 1538443239 Oct 01 18:24:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 01 18:30:16 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 18:30:16 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 01 18:31:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 18:42:44 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 18:42:44 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 01 18:52:44 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 18:52:44 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 01 19:05:48 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 19:05:48 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Oct 01 19:15:51 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 19:15:51 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages Oct 01 19:28:02 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 19:28:02 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 53 previous similar messages Oct 01 19:38:49 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 19:38:49 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17 previous similar messages Oct 01 19:50:45 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 19:50:45 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17 previous similar messages Oct 01 19:57:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 01 19:57:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 01 19:57:59 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 01 20:09:23 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 20:09:23 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 01 20:33:03 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 20:42:22 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 20:42:22 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 01 20:52:59 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 20:52:59 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Oct 01 20:59:16 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 20:59:16 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 01 21:14:11 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 21:14:11 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 24 previous similar messages Oct 01 21:31:09 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 21:31:09 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 01 21:41:38 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 21:41:38 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 01 21:53:27 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 21:53:27 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 01 22:09:25 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 22:09:25 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages Oct 01 22:19:50 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 22:19:50 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 01 22:29:58 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 22:29:58 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 43 previous similar messages Oct 01 22:40:32 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 22:40:32 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 45 previous similar messages Oct 01 22:47:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f35ebc7-dc64-a998-8312-2cf35186f7db (at 10.8.6.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880052c80800, cur 1538459238 expire 1538459088 last 1538459011 Oct 01 22:57:51 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 22:57:51 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 01 23:14:23 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 23:14:23 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Oct 01 23:39:15 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 23:39:15 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 29 previous similar messages Oct 01 23:40:37 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 23:44:07 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 23:44:07 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Oct 01 23:53:02 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 01 23:53:02 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 02 00:20:21 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 00:20:21 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 02 00:40:42 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 00:40:42 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 02 00:42:52 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 00:42:52 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 02 00:45:12 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 00:45:12 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 02 00:48:55 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 00:51:44 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 00:51:44 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 02 01:00:42 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 01:00:42 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14 previous similar messages Oct 02 01:06:22 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 01:06:22 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 02 01:23:12 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 01:23:12 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 02 01:34:46 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 01:34:46 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 02 02:12:32 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 02:12:32 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 02 02:21:13 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 02:21:13 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 02 02:34:20 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 02:36:16 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 02:36:16 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 02 02:39:59 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 02:44:32 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 02:44:32 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 02 02:53:17 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 02:53:17 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 02 03:06:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 02 03:06:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 02 03:16:18 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 03:16:18 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 02 03:17:46 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 03:17:46 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 02 03:18:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 02 03:18:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 02 03:21:12 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 03:21:12 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 02 03:27:19 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 03:27:19 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 02 03:39:48 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 03:39:48 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Oct 02 03:43:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 02 03:43:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 02 03:53:39 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 03:53:39 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 02 04:09:08 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 04:09:08 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 33 previous similar messages Oct 02 04:20:13 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 04:20:13 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 894 previous similar messages Oct 02 04:29:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 02 04:29:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 02 04:30:13 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 04:30:13 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 558 previous similar messages Oct 02 04:42:49 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 04:42:49 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1125 previous similar messages Oct 02 04:53:02 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 04:53:02 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4184 previous similar messages Oct 02 05:05:56 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 05:05:56 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3135 previous similar messages Oct 02 05:16:30 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 05:16:30 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1473 previous similar messages Oct 02 05:26:56 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 05:26:56 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 717 previous similar messages Oct 02 05:39:54 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 05:39:54 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 677 previous similar messages Oct 02 05:51:34 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 05:51:34 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2502 previous similar messages Oct 02 06:02:20 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 06:02:20 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1506 previous similar messages Oct 02 06:14:18 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 06:14:18 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1065 previous similar messages Oct 02 06:24:45 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 06:24:45 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 735 previous similar messages Oct 02 06:35:13 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 06:35:13 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1576 previous similar messages Oct 02 06:45:24 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 06:45:24 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4834 previous similar messages Oct 02 06:57:34 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 06:57:34 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1648 previous similar messages Oct 02 07:07:35 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 07:07:35 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2408 previous similar messages Oct 02 07:19:00 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 07:19:00 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2395 previous similar messages Oct 02 07:29:51 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 07:29:51 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4986 previous similar messages Oct 02 07:40:30 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 07:40:30 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2382 previous similar messages Oct 02 07:50:45 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 07:50:45 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2123 previous similar messages Oct 02 08:01:47 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 08:01:47 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2923 previous similar messages Oct 02 08:13:09 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 08:13:09 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2182 previous similar messages Oct 02 08:23:14 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 08:23:14 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2890 previous similar messages Oct 02 08:33:16 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 08:33:16 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4060 previous similar messages Oct 02 08:43:24 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 08:43:24 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1537 previous similar messages Oct 02 08:55:54 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 08:55:54 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1606 previous similar messages Oct 02 09:05:57 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 09:05:57 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1578 previous similar messages Oct 02 09:16:20 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 09:16:20 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2992 previous similar messages Oct 02 09:29:40 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 09:29:40 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2899 previous similar messages Oct 02 09:31:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 26236fa6-1d12-de32-38f2-2dd2f2520427 (at 10.12.4.80@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805adfadc00, cur 1538497873 expire 1538497723 last 1538497646 Oct 02 09:39:45 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 09:39:45 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1008 previous similar messages Oct 02 09:49:50 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 09:49:50 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2377 previous similar messages Oct 02 10:01:55 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 10:01:55 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2338 previous similar messages Oct 02 10:12:09 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 10:12:09 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1696 previous similar messages Oct 02 10:22:10 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 10:22:10 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1242 previous similar messages Oct 02 10:32:54 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 10:32:54 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1690 previous similar messages Oct 02 10:44:39 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 10:44:39 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2247 previous similar messages Oct 02 10:55:22 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 10:55:22 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 348 previous similar messages Oct 02 11:08:22 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 11:08:22 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 456 previous similar messages Oct 02 11:18:54 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 11:18:54 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 710 previous similar messages Oct 02 11:30:19 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 11:30:19 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1146 previous similar messages Oct 02 11:40:19 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 11:40:19 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2503 previous similar messages Oct 02 11:51:23 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 11:51:23 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2186 previous similar messages Oct 02 12:01:24 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 12:01:24 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1252 previous similar messages Oct 02 12:11:24 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 12:11:24 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1369 previous similar messages Oct 02 12:21:31 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 12:21:31 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2017 previous similar messages Oct 02 12:24:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a56813ee-c4bd-68c7-4cc9-53dc1806e389 (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e52859800, cur 1538508273 expire 1538508123 last 1538508046 Oct 02 12:32:27 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 12:32:27 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3631 previous similar messages Oct 02 12:42:48 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 12:42:48 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3261 previous similar messages Oct 02 12:44:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ff0677b8-665a-6aca-62c8-0e6ce35ed2ab (at 10.9.113.4@o2ib4) Oct 02 12:54:41 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 12:54:41 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5552 previous similar messages Oct 02 13:04:44 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 13:04:44 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2507 previous similar messages Oct 02 13:14:47 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 13:14:47 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 589 previous similar messages Oct 02 13:27:05 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 13:27:05 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Oct 02 13:31:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0822afea-569d-1eb6-8f64-57449413c992 (at 10.12.4.47@o2ib) Oct 02 13:33:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ac7dd2e6-957c-7138-459f-342637e2dd2b (at 10.12.4.58@o2ib) Oct 02 13:42:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb3886e9-10e1-4513-e59c-fa9ca78b8bf8 (at 10.12.4.83@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88029d91d400, cur 1538512941 expire 1538512791 last 1538512714 Oct 02 13:43:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 468934c8-bd25-066c-7ba5-fc8ddaac13d0 (at 10.12.4.80@o2ib) Oct 02 13:43:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acdd75fe-b325-54ad-3e7e-2cce9f7641e3 (at 10.12.4.83@o2ib) Oct 02 13:51:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 791512df-23bb-2d6d-6dab-915917d0c243 (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809591ce800, cur 1538513476 expire 1538513326 last 1538513249 Oct 02 13:52:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 02 13:52:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 14:52:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51ce3a0c-94fe-ee80-08b3-7b581d1235bc (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88189aa88400, cur 1538517178 expire 1538517028 last 1538516951 Oct 02 15:06:12 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 15:06:12 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 32 previous similar messages Oct 02 15:20:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) Oct 02 16:49:03 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 02 16:50:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 02 16:50:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 02 16:51:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 02 16:51:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 02 16:55:04 oak-md1-s2 kernel: LNet: 154499:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.245@o2ib5 version 12/12 incarnation 1537228912189410/1538524503468256 Oct 02 16:55:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fccddc08-d568-f4b4-6a44-f4e21b04be63 (at 10.0.2.245@o2ib5) Oct 02 16:57:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6bae321f-b6fa-9d73-1053-e986a98d5731 (at 10.0.2.245@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ce878ac00, cur 1538524623 expire 1538524473 last 1538524396 Oct 02 17:50:07 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538527800/real 1538527800] req@ffff880073cf3000 x1610629432693280/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538527807 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 02 17:50:14 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538527807/real 1538527807] req@ffff880073cf3000 x1610629432693280/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538527814 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 02 17:50:21 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538527814/real 1538527814] req@ffff880073cf3000 x1610629432693280/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538527821 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 02 17:50:28 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538527821/real 1538527821] req@ffff880073cf3000 x1610629432693280/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538527828 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 02 17:50:36 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538527828/real 1538527828] req@ffff880073cf3000 x1610629432693280/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538527835 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 02 17:50:50 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538527843/real 1538527843] req@ffff880073cf3000 x1610629432693280/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538527850 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 02 17:50:50 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 02 17:51:11 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538527864/real 1538527864] req@ffff880073cf3000 x1610629432693280/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538527871 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 02 17:51:11 oak-md1-s2 kernel: Lustre: 10563:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 02 17:51:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 042b49f3-d89e-1bc4-fd6b-8f2c97c17daa (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018dd77400, cur 1538527895 expire 1538527745 last 1538527668 Oct 02 17:59:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ab5440d-7d80-2892-274b-1bfaa07b4efd (at 10.9.106.30@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809331bc800, cur 1538528365 expire 1538528215 last 1538528138 Oct 02 18:08:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce0ee106-9626-c6d4-b79a-a8c296fd07dd (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005c333c00, cur 1538528920 expire 1538528770 last 1538528693 Oct 02 18:20:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 02 18:28:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3854a30c-b7f9-a9e9-803c-4783bb383ba0 (at 10.9.106.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c83fbec00, cur 1538530122 expire 1538529972 last 1538529895 Oct 02 18:36:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: 3809:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538534187/real 0] req@ffff881c88364500 x1610629436590320/t0(0) o13->oak-OST002a-osc-MDT0000@10.0.2.101@o2ib5:7/4 lens 224/368 e 0 to 1 dl 1538534194 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: 3809:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: oak-OST002a-osc-MDT0000: Connection to oak-OST002a (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c2068c10-7c4f-34b2-dd40-6bd812b10ffd (at 10.9.102.26@o2ib4) reconnecting Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 156573be-eca5-3a48-3b0f-a12ef5bbd1ab (at 10.9.102.26@o2ib4) Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: oak-OST002d-osc-MDT0000: Connection to oak-OST002d (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 02 19:36:34 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 02 19:36:35 oak-md1-s2 kernel: Lustre: oak-OST001d-osc-MDT0000: Connection to oak-OST001d (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 02 19:36:35 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 02 19:36:38 oak-md1-s2 kernel: Lustre: oak-OST0010-osc-MDT0000: Connection to oak-OST0010 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 02 19:36:38 oak-md1-s2 kernel: Lustre: Skipped 26 previous similar messages Oct 02 19:36:38 oak-md1-s2 kernel: Lustre: 3814:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538534191/real 0] req@ffff88124da3ec00 x1610629436591488/t0(0) o13->oak-OST0027-osc-MDT0000@10.0.2.102@o2ib5:7/4 lens 224/368 e 0 to 1 dl 1538534198 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Oct 02 19:36:38 oak-md1-s2 kernel: Lustre: 3814:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Oct 02 19:36:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client d3f3c223-b8c0-c770-b315-ba2fe2aa243f (at 10.9.101.54@o2ib4) reconnecting Oct 02 19:36:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbb83184-3e78-df7b-5109-069e289706b1 (at 10.9.101.54@o2ib4) Oct 02 19:36:46 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538534200/real 0] req@ffff881a3afb4800 x1610629436593712/t0(0) o8->oak-OST002e-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538534206 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 02 19:36:46 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 57 previous similar messages Oct 02 19:36:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 3ef05dff-086a-a255-0efb-a2b070ec534d (at 10.9.106.18@o2ib4) reconnecting Oct 02 19:36:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2b013eda-077a-097a-0fba-6687e4b3366b (at 10.9.106.18@o2ib4) Oct 02 19:37:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c87d5c6e-a1ff-65da-02c3-e478867f2312 (at 10.9.107.23@o2ib4) reconnecting Oct 02 19:37:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d294355-5de1-5391-a51f-91e6f8ce1ccb (at 10.9.107.23@o2ib4) Oct 02 19:37:05 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538534225/real 1538534225] req@ffff881a3afb5a00 x1610629436597824/t0(0) o8->oak-OST000f-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538534231 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 19:37:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client df6d031e-1200-ab54-995b-2c51eceef136 (at 10.9.104.59@o2ib4) reconnecting Oct 02 19:37:08 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 02 19:37:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e0a2db5a-a04f-2beb-7817-bdc136c7d38c (at 10.9.104.59@o2ib4) Oct 02 19:37:08 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 02 19:37:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8d2ad060-dcd0-ec02-5ab7-993da73cbcbd (at 10.9.102.66@o2ib4) reconnecting Oct 02 19:37:17 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 02 19:37:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8d2ad060-dcd0-ec02-5ab7-993da73cbcbd (at 10.9.102.66@o2ib4) Oct 02 19:37:17 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 02 19:37:33 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.28.7@o2ib6 Oct 02 19:37:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ef65363c-e35a-467c-e258-f6481cb61a57 (at 10.9.102.70@o2ib4) reconnecting Oct 02 19:37:34 oak-md1-s2 kernel: Lustre: Skipped 185 previous similar messages Oct 02 19:37:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ef65363c-e35a-467c-e258-f6481cb61a57 (at 10.9.102.70@o2ib4) Oct 02 19:37:34 oak-md1-s2 kernel: Lustre: Skipped 183 previous similar messages Oct 02 19:37:55 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538534275/real 1538534275] req@ffff881686fe8000 x1610629436606880/t0(0) o8->oak-OST0003-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538534286 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 19:37:55 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 95 previous similar messages Oct 02 19:37:58 oak-md1-s2 kernel: LNetError: 10570:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.28.7@o2ib6 Oct 02 19:38:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2e9cd6f0-b51f-4e21-818c-b26592c41e6d (at 10.9.105.7@o2ib4) reconnecting Oct 02 19:38:06 oak-md1-s2 kernel: Lustre: Skipped 102 previous similar messages Oct 02 19:38:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 44b1a1e5-37ba-7a23-b409-18d01f89c7f8 (at 10.9.105.7@o2ib4) Oct 02 19:38:06 oak-md1-s2 kernel: Lustre: Skipped 102 previous similar messages Oct 02 19:38:45 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.6.36@o2ib6 Oct 02 19:38:47 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.6.36@o2ib6 Oct 02 19:38:47 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) Skipped 574856 previous similar messages Oct 02 19:38:51 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.6.36@o2ib6 Oct 02 19:38:51 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) Skipped 1157576 previous similar messages Oct 02 19:38:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3ca23783-4e9f-0334-0a6d-4b4e7054cbd8 (at 10.12.4.36@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200bed2800, cur 1538534339 expire 1538534189 last 1538534112 Oct 02 19:38:59 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.6.36@o2ib6 Oct 02 19:38:59 oak-md1-s2 kernel: LNetError: 10562:0:(lib-move.c:1557:lnet_select_pathway()) Skipped 2292520 previous similar messages Oct 02 19:38:59 oak-md1-s2 kernel: Lustre: 10562:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538534339/real 1538534339] req@ffff881671fc9500 x1610629436616128/t0(0) o104->oak-MDT0000@10.8.6.36@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538534346 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 Oct 02 19:38:59 oak-md1-s2 kernel: Lustre: 10562:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4036065 previous similar messages Oct 02 19:39:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5025dce0-023b-4b33-0eef-9f835ed1fe42 (at 10.9.103.10@o2ib4) Oct 02 19:39:11 oak-md1-s2 kernel: Lustre: Skipped 86 previous similar messages Oct 02 19:39:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 9f955755-d391-66d1-f722-6ec6b0490b32 (at 10.9.103.15@o2ib4) reconnecting Oct 02 19:39:13 oak-md1-s2 kernel: Lustre: Skipped 75 previous similar messages Oct 02 19:39:38 oak-md1-s2 kernel: LNetError: 10519:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.28.7@o2ib6 Oct 02 19:39:38 oak-md1-s2 kernel: LNetError: 10519:0:(lib-move.c:1557:lnet_select_pathway()) Skipped 95078 previous similar messages Oct 02 19:40:28 oak-md1-s2 kernel: LNetError: 10586:0:(lib-move.c:1557:lnet_select_pathway()) no route to 10.8.28.7@o2ib6 Oct 02 19:41:01 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 0 seconds Oct 02 19:41:01 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 20 previous similar messages Oct 02 19:41:21 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538534450/real 0] req@ffff8815e5e39e00 x1610629436639728/t0(0) o8->oak-OST002f-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538534481 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 02 19:41:21 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 84302 previous similar messages Oct 02 19:41:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 06e65363-e6fb-6d44-60b7-3bfdff68cdc6 (at 10.9.0.64@o2ib4) Oct 02 19:41:24 oak-md1-s2 kernel: Lustre: Skipped 77 previous similar messages Oct 02 19:41:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c2068c10-7c4f-34b2-dd40-6bd812b10ffd (at 10.9.102.26@o2ib4) reconnecting Oct 02 19:41:45 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 02 19:42:17 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 11 seconds Oct 02 19:42:17 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 49 previous similar messages Oct 02 19:44:36 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 0 seconds Oct 02 19:44:36 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 131 previous similar messages Oct 02 19:45:39 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538534725/real 1538534739] req@ffff881b8367e000 x1610629436685920/t0(0) o8->oak-OST000d-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538534766 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 19:45:39 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 231 previous similar messages Oct 02 19:46:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b75866aa-418d-e1f3-5cbd-48d4dc354fb1 (at 10.8.10.9@o2ib6) Oct 02 19:46:01 oak-md1-s2 kernel: Lustre: Skipped 37 previous similar messages Oct 02 19:47:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8794f9ed-71b5-361a-612d-721de1ee6a63 (at 10.8.28.7@o2ib6) reconnecting Oct 02 19:47:08 oak-md1-s2 kernel: Lustre: Skipped 35 previous similar messages Oct 02 19:49:01 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 265 seconds Oct 02 19:49:01 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 299 previous similar messages Oct 02 19:53:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ce907ce-b30d-2e0c-ae2c-1cf7860ea00c (at 10.8.21.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af4800, cur 1538535188 expire 1538535038 last 1538534961 Oct 02 19:53:08 oak-md1-s2 kernel: Lustre: Skipped 932 previous similar messages Oct 02 19:54:15 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538535200/real 0] req@ffff881c1eec7b00 x1610629436767328/t0(0) o8->oak-OST0027-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538535255 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 02 19:54:15 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 500 previous similar messages Oct 02 19:54:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 78aa76b6-acfb-8a3d-07a7-6cf45e8ca312 (at 10.8.28.9@o2ib6) in 189 seconds. I think it's dead, and I am evicting it. exp ffff88010988e000, cur 1538535264 expire 1538535114 last 1538535075 Oct 02 19:54:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 02 19:57:38 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 34 seconds Oct 02 19:57:38 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 449 previous similar messages Oct 02 20:00:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6bfed4e-8ef8-7430-8dd5-bdfc78ab5fc5 (at 10.8.7.30@o2ib6) Oct 02 20:00:45 oak-md1-s2 kernel: Lustre: Skipped 1240 previous similar messages Oct 02 20:04:22 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538535825/real 1538535862] req@ffff881b84a41b00 x1610629436872496/t0(0) o8->oak-OST001a-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538535880 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 20:04:22 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 528 previous similar messages Oct 02 20:07:44 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 213 seconds Oct 02 20:07:44 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 589 previous similar messages Oct 02 20:14:28 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538536450/real 1538536468] req@ffff88188628c500 x1610629436979792/t0(0) o8->oak-OST0005-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538536505 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 20:14:28 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 656 previous similar messages Oct 02 20:17:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71511167-f77d-1aa1-62d5-41fa439bc8a7 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185bae9c00, cur 1538536662 expire 1538536512 last 1538536435 Oct 02 20:17:50 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 58 seconds Oct 02 20:17:50 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 589 previous similar messages Oct 02 20:18:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 02 20:24:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb3bf895-ccdd-d9d8-7200-4816c0f4bf6c (at 10.8.29.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28d400, cur 1538537063 expire 1538536913 last 1538536836 Oct 02 20:24:34 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538537025/real 1538537074] req@ffff88183df60900 x1610629437076880/t0(0) o8->oak-OST002b-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538537080 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 20:24:34 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 512 previous similar messages Oct 02 20:27:56 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 50 seconds Oct 02 20:27:56 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 583 previous similar messages Oct 02 20:30:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46fae0d7-e612-0ffd-26c1-dfb42003d51a (at 10.8.1.8@o2ib6) Oct 02 20:30:23 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 02 20:31:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28183dad-f8b0-8593-0243-0b36c3c2d2b9 (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bee61e400, cur 1538537462 expire 1538537312 last 1538537235 Oct 02 20:34:40 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538537650/real 1538537680] req@ffff881495ec3000 x1610629437187360/t0(0) o8->oak-OST002d-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538537705 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 20:34:40 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 751 previous similar messages Oct 02 20:35:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 46b57932-f635-d0cf-ce9c-79aedcde3868 (at 10.8.18.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185ff55400, cur 1538537740 expire 1538537590 last 1538537513 Oct 02 20:36:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d858ebc6-363f-c492-ad4b-3ae67e2325ec (at 10.9.104.35@o2ib4) in 198 seconds. I think it's dead, and I am evicting it. exp ffff881e61526c00, cur 1538537816 expire 1538537666 last 1538537618 Oct 02 20:36:56 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Oct 02 20:38:02 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 656 seconds Oct 02 20:38:02 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 767 previous similar messages Oct 02 20:44:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8923c5f1-968a-4153-6d4b-2ff3fe2abbf1 (at 10.8.2.22@o2ib6) Oct 02 20:44:18 oak-md1-s2 kernel: Lustre: Skipped 51 previous similar messages Oct 02 20:44:46 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538538250/real 1538538286] req@ffff881c58e12d00 x1610629437292320/t0(0) o8->oak-OST001b-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538538305 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 20:44:46 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 767 previous similar messages Oct 02 20:48:08 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 1262 seconds Oct 02 20:48:08 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 767 previous similar messages Oct 02 20:54:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c10200f2-f14f-e8ae-df54-abd162920c05 (at 10.9.107.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881888aa7400, cur 1538538856 expire 1538538706 last 1538538629 Oct 02 20:54:16 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 02 20:55:00 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538538900/real 1538538900] req@ffff881844adb600 x1610629437408432/t0(0) o8->oak-OST0003-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538538955 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 20:55:00 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 977 previous similar messages Oct 02 20:58:14 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 32 seconds Oct 02 20:58:14 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 119 previous similar messages Oct 02 20:58:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 498597ee-7d0c-a101-0428-779bd5f7462d (at 10.9.105.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d2be1ac00, cur 1538539097 expire 1538538947 last 1538538870 Oct 02 20:59:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2455eb3f-12df-a8cc-df9f-a0917ba1883c (at 10.8.1.18@o2ib6) Oct 02 20:59:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 21:05:11 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538539475/real 1538539511] req@ffff881800afc200 x1610629437509632/t0(0) o8->oak-OST0029-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538539530 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 21:05:11 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 750 previous similar messages Oct 02 21:06:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 621db5eb-e81b-8c40-b02a-70a8bffc2911 (at 10.9.101.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88120ee8f800, cur 1538539573 expire 1538539423 last 1538539346 Oct 02 21:08:20 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 76 seconds Oct 02 21:08:20 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 689 previous similar messages Oct 02 21:11:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 128c8ccd-b8b6-6943-4767-b0b8fbf966c4 (at 10.8.26.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8cd800, cur 1538539884 expire 1538539734 last 1538539657 Oct 02 21:12:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea396b48-0df5-4bd1-490c-f8dd9680665b (at 10.12.4.55@o2ib) in 186 seconds. I think it's dead, and I am evicting it. exp ffff88092a989800, cur 1538539960 expire 1538539810 last 1538539774 Oct 02 21:15:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 003bfd88-919c-3745-858b-eadb4f8ad9d1 (at 10.12.4.55@o2ib) Oct 02 21:15:06 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 02 21:15:17 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538540100/real 1538540117] req@ffff881cb235a700 x1610629437616928/t0(0) o8->oak-OST0018-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538540155 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 21:15:17 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 656 previous similar messages Oct 02 21:18:26 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 80 seconds Oct 02 21:18:26 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 642 previous similar messages Oct 02 21:20:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a90f8dd8-643f-7a60-7b29-c54bdb9a9464 (at 10.8.22.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea512f000, cur 1538540411 expire 1538540261 last 1538540184 Oct 02 21:21:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e3ffe529-a2e4-8a2a-6dea-c640b3504a5f (at 10.8.30.7@o2ib6) in 224 seconds. I think it's dead, and I am evicting it. exp ffff880d773e4000, cur 1538540487 expire 1538540337 last 1538540263 Oct 02 21:21:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 21:25:23 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538540675/real 1538540723] req@ffff88154b271800 x1610629437714896/t0(0) o8->oak-OST0007-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538540730 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 21:25:23 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 584 previous similar messages Oct 02 21:28:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9177b321-2ac7-2ddc-677a-af070454bbdc (at 10.9.105.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b76fbec00, cur 1538540897 expire 1538540747 last 1538540670 Oct 02 21:28:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 21:28:32 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 87 seconds Oct 02 21:28:32 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 517 previous similar messages Oct 02 21:29:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5158eccf-14ae-5d94-c749-4edeb98a44a2 (at 10.8.6.8@o2ib6) Oct 02 21:29:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 02 21:34:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 64b6a657-6639-43a8-0ece-0128ae2feca2 (at 10.8.11.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005dcb5400, cur 1538541262 expire 1538541112 last 1538541035 Oct 02 21:35:29 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538541300/real 1538541329] req@ffff881b73a91500 x1610629437823040/t0(0) o8->oak-OST0002-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538541355 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 21:35:29 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 596 previous similar messages Oct 02 21:38:38 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 56 seconds Oct 02 21:38:38 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 447 previous similar messages Oct 02 21:39:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e36ee58-9852-8867-1788-cd0ba4d0b6af (at 10.9.103.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001dd4c00, cur 1538541577 expire 1538541427 last 1538541350 Oct 02 21:39:37 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 02 21:41:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 011b86fc-f26a-5df7-ccc5-65a05ffcd821 (at 10.8.1.6@o2ib6) Oct 02 21:41:40 oak-md1-s2 kernel: Lustre: Skipped 31 previous similar messages Oct 02 21:45:30 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538541875/real 0] req@ffff881cab39b900 x1610629437919760/t0(0) o8->oak-OST001c-osc-MDT0000@10.0.2.101@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538541930 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 02 21:45:30 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 491 previous similar messages Oct 02 21:48:44 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.101@o2ib5: 51 seconds Oct 02 21:48:44 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 507 previous similar messages Oct 02 21:48:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 96e45bca-0891-cecc-a067-bff371a41271 (at 10.8.22.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003a19ec00, cur 1538542138 expire 1538541988 last 1538541911 Oct 02 21:48:58 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 02 21:53:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ae47938c-b6df-b044-01ea-ae19446badae (at 10.9.107.28@o2ib4) reconnecting Oct 02 21:53:51 oak-md1-s2 kernel: Lustre: Skipped 663 previous similar messages Oct 02 21:53:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49e1cecb-7d23-0c9c-4906-aea4cc82cdfe (at 10.9.107.28@o2ib4) Oct 02 21:53:51 oak-md1-s2 kernel: Lustre: Skipped 26 previous similar messages Oct 02 21:54:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 57159387-3baf-046b-f0af-f78da67eb6e3 (at 10.8.26.13@o2ib6) reconnecting Oct 02 21:54:55 oak-md1-s2 kernel: Lustre: Skipped 308 previous similar messages Oct 02 21:55:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:55:08 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Rejecting reconnect from the known client oak-MDT0000-lwp-OST002c_UUID (at 10.0.2.101@o2ib5) because it is indicating it is a new client Oct 02 21:55:09 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:55:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 21:55:09 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Rejecting reconnect from the known client oak-MDT0000-lwp-OST0018_UUID (at 10.0.2.101@o2ib5) because it is indicating it is a new client Oct 02 21:55:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 21:55:10 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:10 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:55:11 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 02 21:55:11 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Rejecting reconnect from the known client oak-MDT0000-lwp-OST0008_UUID (at 10.0.2.101@o2ib5) because it is indicating it is a new client Oct 02 21:55:11 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 02 21:55:11 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:12 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:12 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:55:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 02 21:55:13 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Rejecting reconnect from the known client oak-MDT0000-lwp-OST0020_UUID (at 10.0.2.101@o2ib5) because it is indicating it is a new client Oct 02 21:55:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 02 21:55:14 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:14 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:15 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:15 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:16 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:16 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:17 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:55:17 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 02 21:55:17 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Rejecting reconnect from the known client oak-MDT0000-lwp-OST001c_UUID (at 10.0.2.101@o2ib5) because it is indicating it is a new client Oct 02 21:55:17 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 02 21:55:18 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:19 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:19 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:19 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:20 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:20 oak-md1-s2 kernel: format at ldlm_lib.c:1215:target_handle_connect doesn't end in newline Oct 02 21:55:41 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538542530/real 1538542541] req@ffff881870e08000 x1610629438031632/t0(0) o8->oak-OST0013-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538542585 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 02 21:55:41 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 587 previous similar messages Oct 02 21:56:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:56:10 oak-md1-s2 kernel: Lustre: Skipped 28 previous similar messages Oct 02 21:57:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:57:00 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Oct 02 21:57:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.101@o2ib5, removing former export from same NID Oct 02 21:57:50 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 02 21:57:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.101@o2ib5, removing former export from 10.0.2.102@o2ib5 Oct 02 21:57:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.101@o2ib5, removing former export from 10.0.2.102@o2ib5 Oct 02 21:57:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.101@o2ib5, removing former export from 10.0.2.102@o2ib5 Oct 02 21:57:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 21:57:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.101@o2ib5, removing former export from 10.0.2.102@o2ib5 Oct 02 21:57:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 02 21:58:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received LWP connection from 10.0.2.101@o2ib5, removing former export from 10.0.2.102@o2ib5 Oct 02 21:58:00 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 02 21:59:16 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 1 seconds Oct 02 21:59:16 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 461 previous similar messages Oct 02 22:00:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da80f778-5212-ad09-a3ec-dacb25dca342 (at 10.8.10.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28ec00, cur 1538542838 expire 1538542688 last 1538542611 Oct 02 22:10:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cac9a134-7469-7012-937a-4cda121773f0 (at 10.8.22.27@o2ib6) Oct 02 22:10:30 oak-md1-s2 kernel: Lustre: Skipped 509 previous similar messages Oct 02 22:16:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a0f136f4-b3bd-6ede-c1eb-8fa4bd548ca8 (at 10.12.4.47@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e0800, cur 1538543814 expire 1538543664 last 1538543587 Oct 02 22:16:54 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Oct 02 22:20:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6ead0f85-43d0-3aed-00d2-889ad00df507 (at 10.9.106.51@o2ib4) Oct 02 22:20:45 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 02 22:29:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3ef2c5f1-9a17-2793-8bdb-e5264782ee8a (at 10.9.106.30@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9441000, cur 1538544545 expire 1538544395 last 1538544318 Oct 02 22:30:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 463044d9-8fba-9e68-ba89-ce485724a8da (at 10.8.12.26@o2ib6) Oct 02 22:30:51 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Oct 02 22:42:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d4646600-e100-6c20-a4ee-8d30424b214f (at 10.9.106.30@o2ib4) Oct 02 22:42:39 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 02 22:48:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 02 22:48:44 oak-md1-s2 kernel: Lustre: Skipped 105 previous similar messages Oct 02 22:52:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3b51c53-2744-4aa9-6be6-0535bf574f36 (at 10.8.8.33@o2ib6) Oct 02 22:52:43 oak-md1-s2 kernel: Lustre: Skipped 36 previous similar messages Oct 02 23:02:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f2874060-6bfb-c6c5-c8a8-60d8ca27c3be (at 10.8.6.7@o2ib6) Oct 02 23:02:55 oak-md1-s2 kernel: Lustre: Skipped 92 previous similar messages Oct 02 23:07:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 02 23:10:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ab51735-c315-f976-5e4c-e2a30b525986 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e5800, cur 1538547011 expire 1538546861 last 1538546784 Oct 02 23:10:11 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 02 23:13:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 02 23:13:28 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 02 23:17:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 50737f96-d7cd-d7c6-19fd-5e4f49650b00 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041fb38800, cur 1538547461 expire 1538547311 last 1538547234 Oct 02 23:35:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d02addfd-1d23-3ae8-8dc1-18aa5d4cca28 (at 10.12.4.86@o2ib) Oct 02 23:35:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 02 23:38:16 oak-md1-s2 kernel: Lustre: 7070:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538548689/real 1538548689] req@ffff8800443df500 x1610629471191216/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538548696 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 02 23:38:16 oak-md1-s2 kernel: Lustre: 7070:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 78 previous similar messages Oct 02 23:39:33 oak-md1-s2 kernel: Lustre: 7070:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538548766/real 1538548766] req@ffff8800443df500 x1610629471191216/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538548773 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 02 23:39:33 oak-md1-s2 kernel: Lustre: 7070:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 02 23:40:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d88dc73f-b51e-3965-049d-7628317a3b22 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816d4e04000, cur 1538548817 expire 1538548667 last 1538548590 Oct 02 23:41:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 02 23:48:22 oak-md1-s2 kernel: Lustre: 10610:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538549294/real 1538549294] req@ffff88174da25d00 x1610629475749504/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538549301 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 02 23:48:22 oak-md1-s2 kernel: Lustre: 10610:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Oct 02 23:50:49 oak-md1-s2 kernel: LustreError: 10610:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff88174da25d00 x1610629475749504 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff88183d2a4400/0x6c00c3f96668ebb0 lrc: 4/0,0 mode: PR/PR res: [0x2000174fb:0x88b9:0x0].0x0 bits 0x13 rrc: 41 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0xbedb0b01be5baf62 expref: 76 pid: 10646 timeout: 6833785067 lvb_type: 0 Oct 02 23:50:49 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 02 23:53:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 00:01:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7351d3e4-aad3-6a81-e05c-eabd03828010 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1f6ad000, cur 1538550095 expire 1538549945 last 1538549868 Oct 03 00:09:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f45c3c3-503e-9e90-d4b3-4d97ac13b3bf (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d169c9400, cur 1538550549 expire 1538550399 last 1538550322 Oct 03 00:10:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 00:10:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 00:10:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 00:11:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 716f18ad-1e0c-6d0d-bd7c-fa82b7beac7c (at 10.8.7.24@o2ib6) Oct 03 00:12:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 00:21:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d1dbfb5e-6494-655e-bcd8-685481ee52de (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e88ee6c00, cur 1538551282 expire 1538551132 last 1538551055 Oct 03 00:22:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 00:25:32 oak-md1-s2 kernel: Lustre: 10595:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538551521/real 1538551521] req@ffff8818532d8c00 x1610629482562640/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538551532 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 03 00:25:32 oak-md1-s2 kernel: Lustre: 10595:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Oct 03 00:25:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f2c8543-2ec8-0f17-ca10-9420ae917d10 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185d7d7400, cur 1538551556 expire 1538551406 last 1538551329 Oct 03 00:26:16 oak-md1-s2 kernel: Lustre: 10595:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538551565/real 1538551565] req@ffff8818532d8c00 x1610629482562640/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538551576 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 00:26:16 oak-md1-s2 kernel: Lustre: 10595:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Oct 03 00:27:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97d3fbcf-4fb2-f457-2f15-e34e8a2fcee8 (at 10.8.4.15@o2ib6) in 180 seconds. I think it's dead, and I am evicting it. exp ffff880cc3efc400, cur 1538551632 expire 1538551482 last 1538551452 Oct 03 00:28:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 00:34:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5681caf5-6cc6-6464-0b08-6292624592e1 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c68adf000, cur 1538552054 expire 1538551904 last 1538551827 Oct 03 00:35:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e3194496-dfee-b6b7-fbaa-643c15d78d0c (at 10.8.4.15@o2ib6) in 186 seconds. I think it's dead, and I am evicting it. exp ffff8817eaf01800, cur 1538552130 expire 1538551980 last 1538551944 Oct 03 00:36:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 00:38:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cdea6245-6be6-9da6-19ef-ef8fac3d0536 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af7c00, cur 1538552318 expire 1538552168 last 1538552091 Oct 03 00:39:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 00:39:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 00:41:08 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 00:42:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 82ecc4f5-e418-fc57-d88b-8822461f8151 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c20b7e400, cur 1538552570 expire 1538552420 last 1538552343 Oct 03 00:50:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 211705f2-0b7b-1412-8617-36171744ea92 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881243748000, cur 1538553038 expire 1538552888 last 1538552811 Oct 03 00:51:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 716f18ad-1e0c-6d0d-bd7c-fa82b7beac7c (at 10.8.7.24@o2ib6) Oct 03 00:51:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 00:54:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42fed998-b13e-59f1-cdec-a8651ba9e684 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db0182800, cur 1538553252 expire 1538553102 last 1538553025 Oct 03 00:57:35 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538553448/real 1538553448] req@ffff8814b0a42a00 x1610629488152784/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538553455 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 03 00:57:35 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 03 00:57:49 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538553462/real 1538553462] req@ffff8814b0a42a00 x1610629488152784/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538553469 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 00:57:49 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 03 00:58:11 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538553483/real 1538553483] req@ffff8814b0a42a00 x1610629488152784/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538553490 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 00:58:11 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 03 00:58:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d086931b-30be-0d13-c523-123a35317164 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881906265c00, cur 1538553498 expire 1538553348 last 1538553271 Oct 03 00:58:53 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538553526/real 1538553526] req@ffff8814b0a42a00 x1610629488152784/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538553533 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 00:58:53 oak-md1-s2 kernel: Lustre: 10570:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 03 01:00:03 oak-md1-s2 kernel: LustreError: 10570:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff8814b0a42a00 x1610629488152784 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8814b97aac00/0x6c00c3f9813e9d22 lrc: 4/0,0 mode: PR/PR res: [0x20001704f:0x11b06:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0xe15bc8f7b531120 expref: 78 pid: 4674 timeout: 6837939224 lvb_type: 0 Oct 03 01:00:03 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 03 01:00:27 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 01:09:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5cc7471-4e04-126a-31cf-95c9aa9899cf (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817537dec00, cur 1538554180 expire 1538554030 last 1538553953 Oct 03 01:10:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 01:10:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 01:10:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 01:11:00 oak-md1-s2 kernel: LustreError: 7151:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88018dd76450 x1610630946630896/t0(0) o37->0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4@10.0.2.221@o2ib5:200/0 lens 568/440 e 0 to 0 dl 1538554300 ref 1 fl Interpret:/0/0 rc 0/0 Oct 03 01:15:38 oak-md1-s2 kernel: Lustre: 10557:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538554531/real 1538554531] req@ffff8814c26ebf00 x1610629492855936/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538554538 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 03 01:15:38 oak-md1-s2 kernel: Lustre: 10557:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 03 01:15:52 oak-md1-s2 kernel: Lustre: 10557:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538554545/real 1538554545] req@ffff8814c26ebf00 x1610629492855936/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538554552 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 01:15:52 oak-md1-s2 kernel: Lustre: 10557:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 03 01:16:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c03cc0f1-75ab-a130-6669-b8bfe7361c35 (at 10.8.6.11@o2ib6) reconnecting Oct 03 01:16:13 oak-md1-s2 kernel: Lustre: 10557:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538554566/real 1538554566] req@ffff8814c26ebf00 x1610629492855936/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538554573 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 01:16:13 oak-md1-s2 kernel: Lustre: 10557:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Oct 03 01:16:55 oak-md1-s2 kernel: Lustre: 10642:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538554608/real 1538554608] req@ffff8809b61d3f00 x1610629492856864/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538554615 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 01:16:55 oak-md1-s2 kernel: Lustre: 10642:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 03 01:18:06 oak-md1-s2 kernel: LustreError: 10557:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff8814c26ebf00 x1610629492855936 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880416742400/0x6c00c3f988d1b9a0 lrc: 4/0,0 mode: PR/PR res: [0x20001704f:0x11ed6:0x0].0x0 bits 0x13 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0xafe99a30f36ada82 expref: 79 pid: 10588 timeout: 6839021846 lvb_type: 0 Oct 03 01:18:06 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 03 01:18:06 oak-md1-s2 kernel: LustreError: 10557:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 1 previous similar message Oct 03 01:28:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33c1af0f-36b1-a963-a0cc-bef98c8335f5 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ed2223800, cur 1538555316 expire 1538555166 last 1538555089 Oct 03 01:32:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 01:32:12 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 01:36:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 01:37:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 01:44:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dab705b5-511d-2585-9ea0-86b297c1ecec (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8af9000, cur 1538556270 expire 1538556120 last 1538556043 Oct 03 01:44:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 01:45:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 01:45:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 01:58:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b0220e8-b3b3-8b67-ab8b-66dfc1cf50f6 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003b698000, cur 1538557087 expire 1538556937 last 1538556860 Oct 03 01:58:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 02:13:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5c87ef10-c339-17df-ec60-ae895723638f (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039233c00, cur 1538558024 expire 1538557874 last 1538557797 Oct 03 02:13:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 02:15:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 02:15:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 02:25:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e5dfca1-24d1-78be-3573-212d30f00492 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bbf1de400, cur 1538558737 expire 1538558587 last 1538558510 Oct 03 02:25:37 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 02:27:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 02:27:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 02:30:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 02:38:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ba9baebd-d86c-cc09-6d4a-c0287836d02d (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b2689800, cur 1538559503 expire 1538559353 last 1538559276 Oct 03 02:38:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 02:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 02:38:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 02:47:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 02:51:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c777ecd-ab89-0c86-b1c9-585664ab5236 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184f70b000, cur 1538560288 expire 1538560138 last 1538560061 Oct 03 02:51:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 02:56:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 02:56:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 03:05:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 03:08:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 887ecf30-f8c6-e69e-2b58-9929f79fde62 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802fde2e000, cur 1538561328 expire 1538561178 last 1538561101 Oct 03 03:09:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 03:09:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 03:20:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 03:21:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ea74e89-8c6d-f9d2-8c2f-49c5f428b143 (at 10.8.12.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810302f3c00, cur 1538562073 expire 1538561923 last 1538561846 Oct 03 03:21:13 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 03:21:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 03:32:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2fae8584-2e73-db44-e63d-b10d1a259366 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818893edc00, cur 1538562725 expire 1538562575 last 1538562498 Oct 03 03:32:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 03:37:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 03:37:25 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 03:42:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1dd14dfe-cb21-3fa3-fd28-d8bf52bf25ae (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808cdbddc00, cur 1538563347 expire 1538563197 last 1538563120 Oct 03 03:42:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 03:48:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 03:48:48 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 03:49:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 03:52:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5d3917f4-0d1b-0059-8e60-156947708da1 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b2bc1800, cur 1538563955 expire 1538563805 last 1538563728 Oct 03 04:01:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 716f18ad-1e0c-6d0d-bd7c-fa82b7beac7c (at 10.8.7.24@o2ib6) Oct 03 04:01:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 04:06:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d1e65b4c-dccc-511d-6832-7c56d80b5932 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fddf4c800, cur 1538564818 expire 1538564668 last 1538564591 Oct 03 04:06:58 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 04:15:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 04:15:45 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 04:29:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9f69ce1-3b7f-43e1-c463-dac7be4538ad (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88162be15000, cur 1538566165 expire 1538566015 last 1538565938 Oct 03 04:29:25 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 04:34:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dfd5fda8-5530-3804-c805-85fe6d490ece (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881888aa5c00, cur 1538566473 expire 1538566323 last 1538566246 Oct 03 04:35:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 04:35:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 04:47:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 03 04:47:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) Oct 03 04:54:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4afdbbfc-0f9a-5f14-bf5d-b5cd2ea0b0ab (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef095800, cur 1538567657 expire 1538567507 last 1538567430 Oct 03 04:55:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 03 04:58:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 601ec287-190f-cee7-5b31-84d2e18bafc6 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c64a80400, cur 1538567881 expire 1538567731 last 1538567654 Oct 03 05:03:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 03 05:03:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) Oct 03 05:03:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 05:12:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 05:15:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 03 05:15:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) Oct 03 05:15:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 05:19:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5cedc1ef-1c53-cc2c-c2a3-757d1be963eb (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88120ee8a000, cur 1538569162 expire 1538569012 last 1538568935 Oct 03 05:21:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 03 05:23:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 03 05:27:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94a6f69f-8d9d-9128-f43c-11472d89b3c4 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880854641400, cur 1538569647 expire 1538569497 last 1538569420 Oct 03 05:27:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 05:28:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 716f18ad-1e0c-6d0d-bd7c-fa82b7beac7c (at 10.8.7.24@o2ib6) Oct 03 05:28:27 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 03 05:35:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a3c697a4-cf78-4f22-a5bb-be698163f8c4 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184a667c00, cur 1538570122 expire 1538569972 last 1538569895 Oct 03 05:39:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fc1b54bc-2b19-9826-5d50-08fe946f833d (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814f06d3400, cur 1538570360 expire 1538570210 last 1538570133 Oct 03 05:40:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8eef02c9-eda2-20da-0293-ba4dcc7290ba (at 10.8.4.15@o2ib6) in 223 seconds. I think it's dead, and I am evicting it. exp ffff881aa0f81000, cur 1538570436 expire 1538570286 last 1538570213 Oct 03 05:40:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 05:40:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 05:50:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f3b7a6f-8093-5486-f611-0919675bfa78 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa3f5000, cur 1538571041 expire 1538570891 last 1538570814 Oct 03 05:50:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 05:50:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 05:59:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8db4d8bb-d279-350a-aab5-f9ba21d42ff3 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea36fd000, cur 1538571598 expire 1538571448 last 1538571371 Oct 03 06:05:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 06:18:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6f9fa33-e2e3-5f22-8689-5232557e5b18 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817537dac00, cur 1538572682 expire 1538572532 last 1538572455 Oct 03 06:18:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 06:18:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 06:18:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 06:18:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 06:18:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 06:19:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 06:28:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4953062f-6346-144c-38ff-2adcb99de8d0 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bbafe400, cur 1538573283 expire 1538573133 last 1538573056 Oct 03 06:28:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 06:28:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 06:28:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 06:28:55 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 03 06:39:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71f0ce19-3abf-133a-006b-38826a540062 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e1ff15000, cur 1538573983 expire 1538573833 last 1538573756 Oct 03 06:39:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 06:39:48 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 06:41:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 06:41:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 06:45:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e75838c-f42d-4a2b-d311-f1727c6d32be (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88010988bc00, cur 1538574315 expire 1538574165 last 1538574088 Oct 03 06:50:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 01a3c74b-4f15-90e0-36d5-883d6f8e04c6 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881596264800, cur 1538574650 expire 1538574500 last 1538574423 Oct 03 06:50:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 06:50:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 06:50:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 06:56:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 06:59:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f5838f7-6bfe-454b-a0dc-1d3def4e3309 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef2881000, cur 1538575158 expire 1538575008 last 1538574931 Oct 03 07:03:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 96dd4e3a-5d5e-acae-c57c-ee01be0abcf9 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b2b31400, cur 1538575393 expire 1538575243 last 1538575166 Oct 03 07:03:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 07:03:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 07:06:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9a61058c-1b52-5ba8-6a6d-5656606662fe (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dcdbb8c00, cur 1538575603 expire 1538575453 last 1538575376 Oct 03 07:13:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9eba101e-1947-eeff-b39a-13ea753f5e30 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e45914800, cur 1538576013 expire 1538575863 last 1538575786 Oct 03 07:14:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 07:27:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 23eec8e7-2929-082b-0efd-c1ae823fd5a4 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf1ea1400, cur 1538576847 expire 1538576697 last 1538576620 Oct 03 07:27:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 07:27:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 07:27:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 07:40:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1d4529fa-a880-5882-a7cf-1fd1e1671865 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818987d8c00, cur 1538577657 expire 1538577507 last 1538577430 Oct 03 07:40:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 07:40:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 07:40:59 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 07:55:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cafbcd43-058f-3ee3-0bcc-73d1c4f8edf3 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b99264400, cur 1538578512 expire 1538578362 last 1538578285 Oct 03 07:55:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 08:10:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 08:10:22 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 03 08:17:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b93367a8-61fb-e9c8-3151-5946095e194e (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b7ae4cc00, cur 1538579834 expire 1538579684 last 1538579607 Oct 03 08:17:14 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 03 08:23:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 954ce481-e01b-e118-ae73-a9dab59e2c2a (at 10.9.102.17@o2ib4) Oct 03 08:23:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 08:36:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb9ab4c9-5097-d3b2-4ee2-5abfdd8c925e (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e61521400, cur 1538580999 expire 1538580849 last 1538580772 Oct 03 08:37:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 08:37:43 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 03 08:38:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 03 08:39:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 03 08:39:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3f3188e-396b-195c-7308-82d5259b5300 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88183ab5a400, cur 1538581185 expire 1538581035 last 1538580958 Oct 03 08:41:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f11e2a51-6ae2-a884-d94b-24f952008c79 (at 10.12.4.30@o2ib) in 152 seconds. I think it's dead, and I am evicting it. exp ffff880bfefc1800, cur 1538581261 expire 1538581111 last 1538581109 Oct 03 08:43:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47147620-53d9-897b-ac50-0e03aac67182 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818feaf8c00, cur 1538581416 expire 1538581266 last 1538581189 Oct 03 08:56:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Oct 03 08:56:34 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 03 08:58:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bcee97a-261e-c156-f715-8c15fb04f239 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186c672c00, cur 1538582311 expire 1538582161 last 1538582084 Oct 03 09:07:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7330ce40-ede5-cd03-50bd-136143fa4969 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058c463400, cur 1538582851 expire 1538582701 last 1538582624 Oct 03 09:07:31 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Oct 03 09:08:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 09:08:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 09:11:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a04488b-f562-2f62-2321-8efc56a61864 (at 10.8.7.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808552a1800, cur 1538583079 expire 1538582929 last 1538582852 Oct 03 09:17:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 722e2a0e-67f3-5921-edbb-995c6dd48494 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806d99f3400, cur 1538583478 expire 1538583328 last 1538583251 Oct 03 09:19:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c9099b4e-3665-e155-4108-e5278e5d37c7 (at 10.9.104.62@o2ib4) Oct 03 09:19:11 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Oct 03 09:28:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 158bb2cc-a1f6-20c7-1f1b-7fa930a37a73 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0e0400, cur 1538584133 expire 1538583983 last 1538583906 Oct 03 09:30:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 09:30:13 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 03 09:34:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7457cc6-3444-e9f3-d413-6359554617e2 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf27bc00, cur 1538584440 expire 1538584290 last 1538584213 Oct 03 09:39:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e40affd-1ade-57b5-e9bb-a7fa62b2d2f4 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a1ee32000, cur 1538584788 expire 1538584638 last 1538584561 Oct 03 09:41:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 716f18ad-1e0c-6d0d-bd7c-fa82b7beac7c (at 10.8.7.24@o2ib6) Oct 03 09:41:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 09:50:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 76566e24-3214-7f15-11f6-5bd442949411 (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe9df6400, cur 1538585459 expire 1538585309 last 1538585232 Oct 03 09:50:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 09:53:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 09:53:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 10:04:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef902d1f-184f-99f8-a8a2-7e06fe01a268 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f98c00, cur 1538586254 expire 1538586104 last 1538586027 Oct 03 10:04:14 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 03 10:04:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 03 10:04:51 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 03 10:17:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3599535e-e3a6-6003-3ba7-779c98ad4ee4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88183ae96c00, cur 1538587057 expire 1538586907 last 1538586830 Oct 03 10:17:37 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 10:18:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2479713c-d92a-9a4c-31db-188d0ee65bc5 (at 10.8.11.14@o2ib6) Oct 03 10:18:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 10:28:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5007ce91-c201-d63c-b37e-015655e5811b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf93a400, cur 1538587705 expire 1538587555 last 1538587478 Oct 03 10:28:25 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 03 10:29:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 03 10:29:02 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 03 10:39:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 03 10:39:30 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 03 10:43:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 897c6b06-9fb0-cd03-0516-e43678c3c7bb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b268e400, cur 1538588630 expire 1538588480 last 1538588403 Oct 03 10:43:50 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 03 10:58:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2bb45e8f-eaa9-50f9-ca37-6ced2592e6b6 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582304400, cur 1538589513 expire 1538589363 last 1538589286 Oct 03 10:58:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 10:59:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 10:59:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 11:06:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:08:06 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 11:08:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:09:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:09:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:10:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 03 11:10:29 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 03 11:10:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6dd5b184-7720-6e31-b538-51c38ddff7c6 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000370a400, cur 1538590236 expire 1538590086 last 1538590009 Oct 03 11:10:36 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 11:11:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:12:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:13:03 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 11:13:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:13:25 oak-md1-s2 kernel: LustreError: 7144:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880018a03600 x1610633311371824/t0(0) o37->0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4@10.0.2.221@o2ib5:73/0 lens 568/440 e 0 to 0 dl 1538590413 ref 1 fl Interpret:/0/0 rc 0/0 Oct 03 11:18:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:22:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d4c4f4d-9a43-7989-e7de-a6c4c4eee848 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a45b72000, cur 1538590928 expire 1538590778 last 1538590701 Oct 03 11:22:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 11:31:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 716f18ad-1e0c-6d0d-bd7c-fa82b7beac7c (at 10.8.7.24@o2ib6) Oct 03 11:31:58 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 03 11:34:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1be7b461-91e0-a68a-d776-05312a9686cb (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e61521c00, cur 1538591670 expire 1538591520 last 1538591443 Oct 03 11:41:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 03 11:41:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 11:44:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:44:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 11:44:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9bc438f-9de7-70ac-6df0-2cfba52fc195 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88145768cc00, cur 1538592285 expire 1538592135 last 1538592058 Oct 03 11:44:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 11:45:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:48:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:52:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:52:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 11:52:44 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 03 11:55:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:57:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 11:57:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 11:58:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ab2729d-125a-fd2a-3de1-1abdb3ebf9a5 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880038e8b400, cur 1538593108 expire 1538592958 last 1538592881 Oct 03 11:58:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 12:03:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 12:03:35 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 03 12:08:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ff825daa-53ac-0836-333a-6441ea96bf72 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801530e8c00, cur 1538593718 expire 1538593568 last 1538593491 Oct 03 12:14:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 12:14:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 12:14:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 12:15:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 12:16:31 oak-md1-s2 kernel: LustreError: 7168:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880c6292b900 x1610633678111408/t0(0) o37->0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4@10.0.2.221@o2ib5:123/0 lens 568/440 e 0 to 0 dl 1538594238 ref 1 fl Interpret:/0/0 rc 0/0 Oct 03 12:21:46 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 12:22:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 12:22:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 12:25:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da418ed5-9ca3-ecc5-d77a-b0eed8bf7a42 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d2e2ce400, cur 1538594726 expire 1538594576 last 1538594499 Oct 03 12:26:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 12:26:55 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 03 12:35:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 12:37:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ac8d313-3f3b-d28d-9f31-d589264b8802 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819f3aae800, cur 1538595443 expire 1538595293 last 1538595216 Oct 03 12:37:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 12:38:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 12:38:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 12:44:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 12:44:55 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 12:55:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 12:55:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 12:55:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 13:00:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 13:02:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 13:05:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 13:05:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 13:05:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 13:07:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 13:09:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b19d27e-c16f-607f-9894-458c6d549900 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4a35c800, cur 1538597371 expire 1538597221 last 1538597144 Oct 03 13:09:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 13:14:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 13:16:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 13:16:02 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 13:18:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4756eb53-ea20-2b5b-d852-d897b9228f09 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227cd400, cur 1538597923 expire 1538597773 last 1538597696 Oct 03 13:20:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 13:20:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 13:20:52 oak-md1-s2 kernel: LustreError: 7144:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88079715c200 x1610634141944896/t0(0) o37->0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4@10.0.2.221@o2ib5:169/0 lens 568/440 e 0 to 0 dl 1538598059 ref 1 fl Interpret:/0/0 rc 0/0 Oct 03 13:30:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 13:30:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 13:30:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 13:30:58 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 13:41:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1dce2f33-b1f0-d0fc-7659-756689da70c5 (at 10.9.107.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a66cdcc00, cur 1538599266 expire 1538599116 last 1538599039 Oct 03 13:42:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df8809d6-7843-d802-a96c-c115f01b3acd (at 10.9.107.12@o2ib4) in 203 seconds. I think it's dead, and I am evicting it. exp ffff8817537dd400, cur 1538599342 expire 1538599192 last 1538599139 Oct 03 13:43:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b83d8da8-6525-3a35-b937-834bafc263c2 (at 10.9.107.4@o2ib4) in 189 seconds. I think it's dead, and I am evicting it. exp ffff881a7d331000, cur 1538599418 expire 1538599268 last 1538599229 Oct 03 13:43:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 13:47:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e920523-5dfe-6c37-46dd-a02d42c2f459 (at 10.9.106.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802955b5800, cur 1538599633 expire 1538599483 last 1538599406 Oct 03 13:54:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c847613e-5e51-8c7f-3d82-22cc0f7f4608 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185ff52000, cur 1538600066 expire 1538599916 last 1538599839 Oct 03 13:55:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 13:55:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 14:02:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a8b50ec-2068-8271-c5af-22e663fb8ac3 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8ce400, cur 1538600550 expire 1538600400 last 1538600323 Oct 03 14:02:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 14:08:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 14:15:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ca2bec57-1b7e-6b78-2273-c821842773cd (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882015cf3800, cur 1538601331 expire 1538601181 last 1538601104 Oct 03 14:18:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 03 14:24:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 14:24:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 14:32:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 14:32:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 14:32:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 14:49:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 78430941-e0f8-ea62-9169-8210b3971f81 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880730189400, cur 1538603397 expire 1538603247 last 1538603170 Oct 03 14:49:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 14:51:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 14:51:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 14:51:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 14:53:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 14:59:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 312c8df4-99b1-8d79-0496-c75d496ba3e0 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e6ba4b400, cur 1538603983 expire 1538603833 last 1538603756 Oct 03 14:59:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 15:02:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 15:02:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 15:03:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 15:03:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e2d3c99-8406-8228-a82e-44b2eccc11a1 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880854643000, cur 1538604220 expire 1538604070 last 1538603993 Oct 03 15:03:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 15:07:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 15:07:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 15:13:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71186ed4-1358-1032-caa9-d91e107e45f8 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db0386400, cur 1538604813 expire 1538604663 last 1538604586 Oct 03 15:14:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 15:24:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 15:24:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 15:24:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 18a255bd-a73a-9336-f6b5-88024aa1082a (at 10.12.4.80@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880551973400, cur 1538605444 expire 1538605294 last 1538605217 Oct 03 15:35:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 15:35:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 15:35:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 15:38:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9574c8e7-427f-e245-369e-9e842e43f889 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e385000, cur 1538606331 expire 1538606181 last 1538606104 Oct 03 15:38:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 15:50:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 15:50:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 15:50:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 15:50:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe4d63dc-b7e3-03eb-78f5-44b21545d202 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88081fe4a400, cur 1538607037 expire 1538606887 last 1538606810 Oct 03 15:56:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 16:01:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4f740fb-78ab-025e-56bf-9480184859bc (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea8c41800, cur 1538607684 expire 1538607534 last 1538607457 Oct 03 16:03:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 16:03:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 16:03:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 16:12:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 16:12:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 16:16:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a5e2678-a1f2-2c40-d989-c21ff7f39746 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803352b2400, cur 1538608567 expire 1538608417 last 1538608340 Oct 03 16:17:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 16:17:21 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 03 16:25:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 16:26:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e2aa8ee-666e-a728-cfde-542d7c6d7069 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d169cb800, cur 1538609193 expire 1538609043 last 1538608966 Oct 03 16:30:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 16:30:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 16:35:12 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:35:12 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 03 16:35:13 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:35:13 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Oct 03 16:35:14 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:35:14 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 29 previous similar messages Oct 03 16:35:29 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:35:29 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 25 previous similar messages Oct 03 16:35:36 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:35:36 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 03 16:35:48 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:35:48 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 51 previous similar messages Oct 03 16:36:48 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:36:48 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 62 previous similar messages Oct 03 16:37:37 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:37:37 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 37 previous similar messages Oct 03 16:39:41 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:39:41 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 49 previous similar messages Oct 03 16:39:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b3886b3-64c4-957b-4aab-7588966c33ee (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bef737800, cur 1538609984 expire 1538609834 last 1538609757 Oct 03 16:41:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 16:41:53 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:41:53 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 46 previous similar messages Oct 03 16:46:49 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:46:49 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 509 previous similar messages Oct 03 16:55:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ed68940-bcda-a9a1-1714-364a47d0fbb6 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880819d36800, cur 1538610946 expire 1538610796 last 1538610719 Oct 03 16:56:01 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 16:56:01 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1534 previous similar messages Oct 03 17:01:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 17:06:53 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 17:06:53 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1577 previous similar messages Oct 03 17:11:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a96ca82d-6e80-49f4-aae3-c2f3b2cf917e (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819f3aacc00, cur 1538611875 expire 1538611725 last 1538611648 Oct 03 17:18:00 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 17:18:00 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3028 previous similar messages Oct 03 17:21:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 03 17:21:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 03 17:24:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 17:28:13 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 17:28:13 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4935 previous similar messages Oct 03 17:32:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b627c794-65f0-99b3-aee1-312d839a04ad (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880be64c2800, cur 1538613134 expire 1538612984 last 1538612907 Oct 03 17:36:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 129af61b-55f5-a701-d112-16d56eaf3092 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881860650c00, cur 1538613405 expire 1538613255 last 1538613178 Oct 03 17:38:14 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 17:38:14 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4279 previous similar messages Oct 03 17:41:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 35c9a1ed-c65f-f256-fdea-e6e3518f6965 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c7fa24c00, cur 1538613664 expire 1538613514 last 1538613437 Oct 03 17:48:26 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 17:48:26 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2601 previous similar messages Oct 03 17:50:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4d594562-5784-832f-b19f-81cb8b62fbc9 (at 10.8.14.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28f800, cur 1538614222 expire 1538614072 last 1538613995 Oct 03 17:56:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 03 17:56:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 03 17:57:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 03 17:57:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 03 17:58:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 17:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 03 17:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 03 18:01:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3036236d-d5b7-16cc-6c31-5df151af3467 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817152e2c00, cur 1538614910 expire 1538614760 last 1538614683 Oct 03 18:01:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 03 18:01:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 03 18:04:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 03 18:07:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 03 18:07:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 18:08:14 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 18:08:14 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 862 previous similar messages Oct 03 18:18:46 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 18:18:46 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 455 previous similar messages Oct 03 18:20:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Oct 03 18:20:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 18:22:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 82d58e4c-78d8-82af-ec93-077395d799ca (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e95c9c000, cur 1538616137 expire 1538615987 last 1538615910 Oct 03 18:22:17 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 03 18:29:37 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 18:29:37 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 260 previous similar messages Oct 03 18:40:48 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 18:40:48 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 411 previous similar messages Oct 03 18:50:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ee4122bd-9ee0-27fd-e82c-f456e310709a (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88068aa2bc00, cur 1538617814 expire 1538617664 last 1538617587 Oct 03 18:51:00 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 18:51:00 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 556 previous similar messages Oct 03 18:51:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 18:51:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 19:02:01 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 19:02:01 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 442 previous similar messages Oct 03 19:06:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a644538c-8f26-65de-ef7b-16fface71c04 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a66cde000, cur 1538618817 expire 1538618667 last 1538618590 Oct 03 19:13:14 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 19:13:14 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 609 previous similar messages Oct 03 19:18:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 19:23:24 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 19:23:24 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 531 previous similar messages Oct 03 19:24:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 91ef8ae0-8911-e302-8d5f-a24cd8ca7fdc (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f196e6400, cur 1538619886 expire 1538619736 last 1538619659 Oct 03 19:35:43 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 19:35:43 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 188 previous similar messages Oct 03 19:47:19 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 19:47:19 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 150 previous similar messages Oct 03 19:57:22 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 19:57:22 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 147 previous similar messages Oct 03 20:01:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 20:25:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5c8d701-3fee-f937-6631-e31566f3ce37 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e8245c00, cur 1538623532 expire 1538623382 last 1538623305 Oct 03 20:35:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 20:54:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d4bff38f-83da-fa43-6939-899fd91aa6d7 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa85fc00, cur 1538625252 expire 1538625102 last 1538625025 Oct 03 20:55:33 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 20:55:33 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 310 previous similar messages Oct 03 20:58:02 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 21:10:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 21:16:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd6fa043-2961-eed9-2211-242ddcb47ef7 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bbafa400, cur 1538626565 expire 1538626415 last 1538626338 Oct 03 21:16:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 21:25:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f3690f01-fb1e-d79d-6160-df1f7df5d771 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b33df800, cur 1538627159 expire 1538627009 last 1538626932 Oct 03 21:42:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 21:58:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f93a4b74-5c0b-29a0-e158-0e25a23fd40f (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880343112800, cur 1538629117 expire 1538628967 last 1538628890 Oct 03 22:04:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 22:16:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc597777-a655-53df-494f-76799a1333c6 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210fc00, cur 1538630160 expire 1538630010 last 1538629933 Oct 03 22:25:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 22:39:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 951110e0-609f-c63e-daea-c407a7cbab48 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b643d400, cur 1538631545 expire 1538631395 last 1538631318 Oct 03 22:40:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 22:45:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05faaa97-d558-c7a8-ca87-cac9a37a0670 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee1a02000, cur 1538631907 expire 1538631757 last 1538631680 Oct 03 22:45:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 22:49:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7930d5ab-da17-a33a-8584-8a7b6292c176 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba0000, cur 1538632145 expire 1538631995 last 1538631918 Oct 03 22:49:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 22:53:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 428c3110-c595-aed2-636e-da1274aaba3b (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef100400, cur 1538632434 expire 1538632284 last 1538632207 Oct 03 22:53:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 22:53:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 22:59:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 044083b2-8de6-a742-65dd-579c4338e0da (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca4c00, cur 1538632767 expire 1538632617 last 1538632540 Oct 03 22:59:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 23:03:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4713e175-d67a-1e95-ff61-308c94a90d2b (at 10.9.101.25@o2ib4) Oct 03 23:11:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 192cc2cd-2deb-477c-5514-44e134ad791f (at 10.12.4.35@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f80400, cur 1538633512 expire 1538633362 last 1538633285 Oct 03 23:12:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa5541c6-b49f-97f9-7ebd-59eaa5151787 (at 10.12.4.35@o2ib) Oct 03 23:18:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb86579d-36d3-16e7-1d9f-0cbd9843eae5 (at 10.12.4.78@o2ib) Oct 03 23:18:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9bb48386-3bfe-1971-683a-da51250b5903 (at 10.12.4.78@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882015c1d400, cur 1538633937 expire 1538633787 last 1538633710 Oct 03 23:18:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 23:20:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 23:20:34 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 03 23:20:34 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 03 23:34:03 oak-md1-s2 kernel: Lustre: 10616:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538634836/real 1538634836] req@ffff8818863d3000 x1610630647268704/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538634843 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 03 23:34:03 oak-md1-s2 kernel: Lustre: 10616:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Oct 03 23:34:17 oak-md1-s2 kernel: Lustre: 10616:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538634850/real 1538634850] req@ffff8818863d3000 x1610630647268704/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538634857 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 23:34:17 oak-md1-s2 kernel: Lustre: 10616:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 03 23:34:39 oak-md1-s2 kernel: Lustre: 10616:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538634871/real 1538634871] req@ffff8818863d3000 x1610630647268704/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538634878 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 03 23:34:39 oak-md1-s2 kernel: Lustre: 10616:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 03 23:35:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f2fd04b-6e28-c7d8-e449-ba74811766a1 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68f400, cur 1538634905 expire 1538634755 last 1538634678 Oct 03 23:38:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 23:40:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e2a8024-2779-1c5d-75e4-74cfd77beec7 (at 10.9.101.47@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e61525800, cur 1538635250 expire 1538635100 last 1538635023 Oct 03 23:44:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6203b3ec-8057-f793-018e-9455f590a819 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802a9bbd800, cur 1538635492 expire 1538635342 last 1538635265 Oct 03 23:45:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 03 23:48:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 846fc1b9-709a-8da9-3bf2-df7529cf3fd4 (at 10.9.101.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817eaf00400, cur 1538635690 expire 1538635540 last 1538635463 Oct 03 23:53:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 00173a6c-d1c1-e0ee-ca21-893e11014697 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4a35bc00, cur 1538636000 expire 1538635850 last 1538635773 Oct 03 23:53:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 03 23:54:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb00c1ca-a02c-8822-a1bb-ad0f0429fe24 (at 10.9.101.47@o2ib4) Oct 04 00:01:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e500cf69-f65b-17e9-a5b0-d0f32692a333 (at 10.9.101.24@o2ib4) Oct 04 00:01:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 00:02:39 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538636552/real 1538636552] req@ffff881c8333e300 x1610630647890432/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538636559 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 00:02:39 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 00:02:46 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538636559/real 1538636559] req@ffff881c8333e300 x1610630647890432/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538636566 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 00:03:00 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538636573/real 1538636573] req@ffff881c8333e300 x1610630647890432/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538636580 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 00:03:00 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 00:03:21 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538636594/real 1538636594] req@ffff881c8333e300 x1610630647890432/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538636601 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 00:03:21 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 00:04:04 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538636636/real 1538636636] req@ffff881c8333e300 x1610630647890432/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538636643 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 00:04:04 oak-md1-s2 kernel: Lustre: 10598:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 00:04:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47c6cba1-02d6-d72c-93eb-db49e2493ed9 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca4800, cur 1538636652 expire 1538636502 last 1538636425 Oct 04 00:05:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 00:05:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 00:09:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd653fd0-4233-ed16-665c-7811e59c2750 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880036e54400, cur 1538636954 expire 1538636804 last 1538636727 Oct 04 00:13:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 00:13:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 00:13:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c297a8e-1a8a-397d-567c-55ae975afa18 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880343113000, cur 1538637218 expire 1538637068 last 1538636991 Oct 04 00:19:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cc7f1263-ccb9-5b31-ca5c-9e660d8211ff (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d28643c00, cur 1538637594 expire 1538637444 last 1538637367 Oct 04 00:26:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 08c8d92d-8612-7893-d947-1a1dae09103a (at 10.8.6.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dab760800, cur 1538638016 expire 1538637866 last 1538637789 Oct 04 00:27:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d296a15e-8154-8527-be08-b302d3e42b22 (at 10.8.6.21@o2ib6) Oct 04 00:35:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4da7e9a3-b206-0c92-df01-913486ad02ce (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096d3f1c00, cur 1538638523 expire 1538638373 last 1538638296 Oct 04 00:38:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 00:38:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 00:41:43 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538638892/real 1538638892] req@ffff881be025da00 x1610630648828352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538638903 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 00:41:43 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 00:41:54 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538638903/real 1538638903] req@ffff881be025da00 x1610630648828352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538638914 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 00:42:16 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538638925/real 1538638925] req@ffff881be025da00 x1610630648828352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538638936 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 00:42:16 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 00:43:00 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538638969/real 1538638969] req@ffff881be025da00 x1610630648828352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538638980 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 00:43:00 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 00:44:06 oak-md1-s2 kernel: LustreError: 4678:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff881be025da00 x1610630648828352 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8801fa453000/0x6c00c3fb373a8fb2 lrc: 4/0,0 mode: PR/PR res: [0x20001704f:0x137d0:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0x7e3dc835aa409237 expref: 79 pid: 4881 timeout: 6923378773 lvb_type: 0 Oct 04 00:44:06 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 04 00:44:06 oak-md1-s2 kernel: LustreError: Skipped 1 previous similar message Oct 04 00:48:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5b55bf50-b42b-f830-627c-73633806c6ac (at 10.12.4.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e3c00, cur 1538639339 expire 1538639189 last 1538639112 Oct 04 00:53:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 00:58:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8a363c6e-3867-43d3-1c2b-5d3983d8943d (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff09b9800, cur 1538639898 expire 1538639748 last 1538639671 Oct 04 01:03:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7916118f-5862-53d8-11c1-13e057087a91 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a2f0400, cur 1538640180 expire 1538640030 last 1538639953 Oct 04 01:29:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 01:29:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 01:47:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f95bf75-b297-6b19-3f2d-05bafc8862d6 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807fa9aa800, cur 1538642863 expire 1538642713 last 1538642636 Oct 04 01:56:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a736d0e-88ad-2da2-bd3a-0924118fc383 (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880551972400, cur 1538643398 expire 1538643248 last 1538643171 Oct 04 02:11:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5fc97944-3706-47a5-8c9c-83f5521a7746 (at 10.12.4.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880562b4e000, cur 1538644285 expire 1538644135 last 1538644058 Oct 04 02:12:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 02:21:44 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538644897/real 1538644897] req@ffff881be025da00 x1610630651091344/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538644904 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 02:21:44 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Oct 04 02:21:58 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538644911/real 1538644911] req@ffff881be025da00 x1610630651091344/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538644918 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 02:21:58 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 02:22:19 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538644932/real 1538644932] req@ffff881be025da00 x1610630651091344/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538644939 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 02:22:19 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 02:23:01 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538644974/real 1538644974] req@ffff881be025da00 x1610630651091344/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538644981 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 02:23:01 oak-md1-s2 kernel: Lustre: 4311:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 02:23:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9dc837e5-4f13-8032-5e99-eb6683122913 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881971e30c00, cur 1538645005 expire 1538644855 last 1538644778 Oct 04 02:24:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 02:34:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bece057b-30c7-1cde-4283-026241c64b34 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f98000, cur 1538645657 expire 1538645507 last 1538645430 Oct 04 02:40:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 02:53:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2616679d-f7ee-a510-a926-39e3082a364b (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68c000, cur 1538646803 expire 1538646653 last 1538646576 Oct 04 02:54:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 02:59:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8102c236-29cb-564a-37c9-5141a7ec9a81 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1a800, cur 1538647145 expire 1538646995 last 1538646918 Oct 04 03:17:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 03:21:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da9321cf-180b-58c2-9432-9f195cfe9220 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa3f0400, cur 1538648462 expire 1538648312 last 1538648235 Oct 04 03:27:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 03:34:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7199b81-f71f-2465-738b-15ad58c64ea7 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc230a400, cur 1538649249 expire 1538649099 last 1538649022 Oct 04 03:39:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 03:44:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 756499ec-5b31-c23c-9c95-85ed714abeaf (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001dd4400, cur 1538649851 expire 1538649701 last 1538649624 Oct 04 03:44:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 03:49:39 oak-md1-s2 kernel: Lustre: 10662:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538650168/real 1538650168] req@ffff881525f3e900 x1610630653011568/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538650179 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 03:49:39 oak-md1-s2 kernel: Lustre: 10662:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 03:49:50 oak-md1-s2 kernel: Lustre: 10662:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538650179/real 1538650179] req@ffff881525f3e900 x1610630653011568/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538650190 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 03:50:12 oak-md1-s2 kernel: Lustre: 10662:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538650201/real 1538650201] req@ffff881525f3e900 x1610630653011568/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538650212 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 03:50:12 oak-md1-s2 kernel: Lustre: 10662:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 03:50:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0604bbb2-b946-a73f-45a5-be01210973ab (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881520e6ec00, cur 1538650236 expire 1538650086 last 1538650009 Oct 04 03:56:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 04:02:43 oak-md1-s2 kernel: Lustre: 10645:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538650952/real 1538650952] req@ffff881be025f200 x1610630653298720/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538650963 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 04:02:43 oak-md1-s2 kernel: Lustre: 10645:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 04:02:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a37f67c2-3a1b-a3a5-85ad-f90ea0a5a3cd (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817537dd000, cur 1538650964 expire 1538650814 last 1538650737 Oct 04 04:03:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 04:09:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 29f6a266-3eb5-8ee3-c79a-2219ff7e1f33 (at 10.9.112.2@o2ib4) Oct 04 04:10:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dfaccb56-d027-e553-ad4e-7814e5d963bf (at 10.9.112.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880adae76800, cur 1538651452 expire 1538651302 last 1538651225 Oct 04 04:12:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2110234f-e5b4-3666-e3ac-bb1e63de2f28 (at 10.9.101.55@o2ib4) in 199 seconds. I think it's dead, and I am evicting it. exp ffff8808f53eb800, cur 1538651528 expire 1538651378 last 1538651329 Oct 04 04:18:42 oak-md1-s2 kernel: Lustre: 10499:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538651911/real 1538651911] req@ffff88051ffe8000 x1610630653646560/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538651922 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 04:18:52 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538651921/real 1538651921] req@ffff88098d151e00 x1610630653650352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538651932 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 04:19:03 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538651932/real 1538651932] req@ffff88098d151e00 x1610630653650352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538651943 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:19:03 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 04:19:14 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538651943/real 1538651943] req@ffff88098d151e00 x1610630653650352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538651954 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:19:14 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 04:19:25 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538651954/real 1538651954] req@ffff88098d151e00 x1610630653650352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538651965 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:19:25 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 04:19:36 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538651965/real 1538651965] req@ffff88098d151e00 x1610630653650352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538651976 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:19:36 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 04:19:59 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538651987/real 1538651987] req@ffff88098d151e00 x1610630653650352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538651998 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:19:59 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 04:20:43 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538652032/real 1538652032] req@ffff88098d151e00 x1610630653650352/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538652043 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:20:43 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Oct 04 04:20:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5a90041c-992a-f207-bb89-d17037199681 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88085dc1ac00, cur 1538652058 expire 1538651908 last 1538651831 Oct 04 04:26:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb00c1ca-a02c-8822-a1bb-ad0f0429fe24 (at 10.9.101.47@o2ib4) Oct 04 04:26:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fd226631-ec42-ea48-1eb1-b60097c20b51 (at 10.9.101.55@o2ib4) Oct 04 04:26:56 oak-md1-s2 kernel: LustreError: 10604:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.101.47@o2ib4) returned error from blocking AST (req@ffff881d1564b600 x1610630653828832 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880dd6603800/0x6c00c3fb5f226e26 lrc: 4/0,0 mode: PR/PR res: [0x200017547:0x1c41e:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.101.47@o2ib4 remote: 0x1495d507c841403c expref: 407 pid: 10513 timeout: 6936759119 lvb_type: 0 Oct 04 04:26:56 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.101.47@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Oct 04 04:49:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8637d08e-c488-99b6-e567-295ff68f8921 (at 10.9.101.29@o2ib4) Oct 04 04:50:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7ff28651-4f03-34a5-4c7f-acba9492f756 (at 10.9.101.29@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde18400, cur 1538653809 expire 1538653659 last 1538653582 Oct 04 04:57:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 04:58:49 oak-md1-s2 kernel: Lustre: 10663:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538654318/real 1538654318] req@ffff881be0258f00 x1610630654522768/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538654329 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 04:58:49 oak-md1-s2 kernel: Lustre: 10663:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 04:59:00 oak-md1-s2 kernel: Lustre: 10663:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538654329/real 1538654329] req@ffff881be0258f00 x1610630654522768/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538654340 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:59:22 oak-md1-s2 kernel: Lustre: 10663:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538654351/real 1538654351] req@ffff881be0258f00 x1610630654522768/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538654362 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 04:59:22 oak-md1-s2 kernel: Lustre: 10663:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 05:00:07 oak-md1-s2 kernel: Lustre: 10663:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538654395/real 1538654395] req@ffff881be0258f00 x1610630654522768/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538654406 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 05:00:07 oak-md1-s2 kernel: Lustre: 10663:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 05:01:13 oak-md1-s2 kernel: LustreError: 10663:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff881be0258f00 x1610630654522768 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881c82607200/0x6c00c3fb6102a434 lrc: 4/0,0 mode: PR/PR res: [0x20001704f:0x13d69:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0xf879cf090f88804a expref: 73 pid: 10631 timeout: 6938805035 lvb_type: 0 Oct 04 05:01:13 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 04 05:05:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 05:14:53 oak-md1-s2 kernel: Lustre: 10493:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538655282/real 1538655282] req@ffff881be025a400 x1610630654876960/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538655293 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 05:14:53 oak-md1-s2 kernel: Lustre: 10493:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Oct 04 05:15:04 oak-md1-s2 kernel: Lustre: 10493:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538655293/real 1538655293] req@ffff881be025a400 x1610630654876960/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538655304 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 05:15:24 oak-md1-s2 kernel: Lustre: 4694:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538655313/real 1538655313] req@ffff880d376a3300 x1610630654884688/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538655324 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 05:15:24 oak-md1-s2 kernel: Lustre: 4694:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 05:15:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1dbd5ac3-1e8b-0f7e-730b-89b0858ee2ae (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88905800, cur 1538655346 expire 1538655196 last 1538655119 Oct 04 05:24:53 oak-md1-s2 kernel: Lustre: 10562:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538655886/real 1538655886] req@ffff8811dbb81b00 x1610630655099760/t0(0) o104->oak-MDT0000@10.8.4.31@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538655893 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 05:24:53 oak-md1-s2 kernel: Lustre: 10562:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 05:26:11 oak-md1-s2 kernel: Lustre: 10562:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538655963/real 1538655963] req@ffff8811dbb81b00 x1610630655099760/t0(0) o104->oak-MDT0000@10.8.4.31@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538655970 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 05:26:11 oak-md1-s2 kernel: Lustre: 10562:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 04 05:27:21 oak-md1-s2 kernel: LustreError: 10562:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.31@o2ib6) failed to reply to blocking AST (req@ffff8811dbb81b00 x1610630655099760 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880aceee5800/0x6c00c3fb620c6366 lrc: 4/0,0 mode: PR/PR res: [0x200017a2b:0xd30:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.8.4.31@o2ib6 remote: 0x52f78064bdb2a3ab expref: 176 pid: 4311 timeout: 6940377068 lvb_type: 0 Oct 04 05:27:21 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.31@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 04 05:28:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 189897b2-be25-1d7e-aa38-95ade6914239 (at 10.8.4.31@o2ib6) Oct 04 05:29:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 05:33:37 oak-md1-s2 kernel: Lustre: 10552:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538656406/real 1538656406] req@ffff880e8c5ec500 x1610630655291984/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538656417 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 05:33:37 oak-md1-s2 kernel: Lustre: 10552:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 04 05:35:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79989465-01f4-feca-59ce-5feaf2942c8d (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d2be1c800, cur 1538656505 expire 1538656355 last 1538656278 Oct 04 05:38:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 05:45:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd8ad3d0-fe38-4c23-450a-05ab6560d479 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe774d800, cur 1538657136 expire 1538656986 last 1538656909 Oct 04 05:49:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 06:09:57 oak-md1-s2 kernel: Lustre: 10601:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538658590/real 1538658590] req@ffff8819d6ef3300 x1610630656095024/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538658597 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 06:09:57 oak-md1-s2 kernel: Lustre: 10601:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Oct 04 06:10:38 oak-md1-s2 kernel: Lustre: 10513:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538658631/real 1538658631] req@ffff880040122700 x1610630656102240/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538658638 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 06:10:38 oak-md1-s2 kernel: Lustre: 10513:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Oct 04 06:11:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2695f120-0398-2df6-a516-1b7da062e3d8 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca4cc00, cur 1538658677 expire 1538658527 last 1538658450 Oct 04 06:15:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 06:18:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ecbb11b-2e2e-4e74-f522-6ec8262ea041 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f712400, cur 1538659135 expire 1538658985 last 1538658908 Oct 04 06:19:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 06:22:44 oak-md1-s2 kernel: Lustre: 10537:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538659353/real 1538659353] req@ffff881b737da400 x1610630656370640/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538659364 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 06:22:44 oak-md1-s2 kernel: Lustre: 10537:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Oct 04 06:22:55 oak-md1-s2 kernel: Lustre: 10537:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538659364/real 1538659364] req@ffff881b737da400 x1610630656370640/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538659375 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 06:23:17 oak-md1-s2 kernel: Lustre: 10537:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538659386/real 1538659386] req@ffff881b737da400 x1610630656370640/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538659397 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 06:23:17 oak-md1-s2 kernel: Lustre: 10537:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 06:24:01 oak-md1-s2 kernel: Lustre: 10537:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538659430/real 1538659430] req@ffff881b737da400 x1610630656370640/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538659441 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 06:24:01 oak-md1-s2 kernel: Lustre: 10537:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 06:25:07 oak-md1-s2 kernel: LustreError: 10537:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff881b737da400 x1610630656370640 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881ea9f67e00/0x6c00c3fb666b7209 lrc: 4/0,0 mode: PR/PR res: [0x200017a2b:0x12e7:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0x48cb6f4ae7843c4b expref: 79 pid: 10578 timeout: 6943839598 lvb_type: 0 Oct 04 06:25:07 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 04 06:29:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 06:48:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a34be9f9-b50c-478f-f363-840eb2a63f86 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811eab25800, cur 1538660912 expire 1538660762 last 1538660685 Oct 04 06:49:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 07:01:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37753d2b-817b-88b3-8565-99c179abdae9 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881812762800, cur 1538661709 expire 1538661559 last 1538661482 Oct 04 07:03:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 07:10:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fb6601ab-39d9-c2d5-b7f0-661993121cb8 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e59b15c00, cur 1538662204 expire 1538662054 last 1538661977 Oct 04 07:31:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 07:46:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 25626bca-aeaa-4a39-2d77-0ee45062463b (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803dd667c00, cur 1538664414 expire 1538664264 last 1538664187 Oct 04 07:48:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 07:57:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client daf2298c-c919-0c09-3efb-8f184508c266 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184d7a9800, cur 1538665032 expire 1538664882 last 1538664805 Oct 04 07:57:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 08:03:04 oak-md1-s2 kernel: Lustre: 10650:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538665373/real 1538665373] req@ffff881b78ace900 x1610630658550832/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538665384 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 08:03:04 oak-md1-s2 kernel: Lustre: 10650:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 04 08:03:15 oak-md1-s2 kernel: Lustre: 10650:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538665384/real 1538665384] req@ffff881b78ace900 x1610630658550832/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538665395 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:03:37 oak-md1-s2 kernel: Lustre: 10650:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538665406/real 1538665406] req@ffff881b78ace900 x1610630658550832/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538665417 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:03:37 oak-md1-s2 kernel: Lustre: 10650:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 08:04:21 oak-md1-s2 kernel: Lustre: 10650:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538665450/real 1538665450] req@ffff881b78ace900 x1610630658550832/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538665461 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:04:21 oak-md1-s2 kernel: Lustre: 10650:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 08:04:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7175b195-d19f-55ac-3b4c-fbcc1af20382 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a30ec00, cur 1538665462 expire 1538665312 last 1538665235 Oct 04 08:21:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 08:28:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0846f68f-a58d-2a7f-f33b-5f0ff07cc68e (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d2692400, cur 1538666898 expire 1538666748 last 1538666671 Oct 04 08:28:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 08:32:48 oak-md1-s2 kernel: Lustre: 10600:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538667157/real 1538667157] req@ffff8811d9678300 x1610630659202176/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538667168 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 08:32:59 oak-md1-s2 kernel: Lustre: 10600:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538667168/real 1538667168] req@ffff8811d9678300 x1610630659202176/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538667179 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:33:21 oak-md1-s2 kernel: Lustre: 10600:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538667190/real 1538667190] req@ffff8811d9678300 x1610630659202176/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538667201 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:33:21 oak-md1-s2 kernel: Lustre: 10600:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 08:34:00 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538667229/real 1538667229] req@ffff8819d6ef0600 x1610630659228720/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538667240 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 08:34:00 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 08:35:11 oak-md1-s2 kernel: LustreError: 10600:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff8811d9678300 x1610630659202176 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8801a7470a00/0x6c00c3fb6c48d2c1 lrc: 4/0,0 mode: PR/PR res: [0x200017547:0x1c8b9:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0x491a624bbe27f33c expref: 79 pid: 10609 timeout: 6951643706 lvb_type: 0 Oct 04 08:35:11 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 04 08:44:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 08:48:27 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538668096/real 1538668096] req@ffff88004de45d00 x1610630659545008/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538668107 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 08:48:27 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Oct 04 08:48:38 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538668107/real 1538668107] req@ffff88004de45d00 x1610630659545008/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538668118 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:49:01 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538668129/real 1538668129] req@ffff88004de45d00 x1610630659545008/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538668140 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:49:01 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 08:49:45 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538668174/real 1538668174] req@ffff88004de45d00 x1610630659545008/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538668185 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 08:49:45 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 08:49:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 057ec834-c667-2227-bab2-3e37cebfd922 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc6c400, cur 1538668187 expire 1538668037 last 1538667960 Oct 04 08:49:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 09:10:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59fd7feb-00c9-e090-8ba6-86d890a1bc70 (at 10.9.101.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817eaf00800, cur 1538669435 expire 1538669285 last 1538669208 Oct 04 09:16:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2d954a7-f735-fe08-792b-a0aca3aa2392 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004b633800, cur 1538669798 expire 1538669648 last 1538669571 Oct 04 09:19:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 09:19:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7e0436e4-3d9b-1e9f-4d08-95b628e10044 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881860650800, cur 1538669955 expire 1538669805 last 1538669728 Oct 04 09:23:49 oak-md1-s2 kernel: Lustre: 10653:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538670218/real 1538670218] req@ffff88080e1c5d00 x1610630660322528/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538670229 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 09:23:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27064999-83c5-825b-e9a8-aafd6dead497 (at 10.9.101.15@o2ib4) Oct 04 09:24:00 oak-md1-s2 kernel: Lustre: 10653:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538670229/real 1538670229] req@ffff88080e1c5d00 x1610630660322528/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538670240 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 09:24:23 oak-md1-s2 kernel: Lustre: 10653:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538670251/real 1538670251] req@ffff88080e1c5d00 x1610630660322528/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538670262 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 09:24:23 oak-md1-s2 kernel: Lustre: 10653:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 09:24:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4bf179b-d387-f312-e595-665838eb3014 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881846f5e800, cur 1538670269 expire 1538670119 last 1538670042 Oct 04 09:27:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 09:31:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0ca98b4-7e4f-4d4f-e3b1-f12600904aaa (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf072400, cur 1538670689 expire 1538670539 last 1538670462 Oct 04 09:32:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f1df0591-6177-86b0-084d-dbfaf6706b71 (at 10.12.4.38@o2ib) in 205 seconds. I think it's dead, and I am evicting it. exp ffff88048a9ee800, cur 1538670765 expire 1538670615 last 1538670560 Oct 04 09:34:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 09:44:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48a1c817-0af9-30b5-a1eb-ce806ac0b702 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803d1770800, cur 1538671454 expire 1538671304 last 1538671227 Oct 04 09:45:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 09:56:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Oct 04 09:58:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b54121f1-dfd1-6fa9-ef90-c65d6798fbb9 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6d1de000, cur 1538672339 expire 1538672189 last 1538672112 Oct 04 10:02:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 214f6e7f-ae81-121f-a9ad-2fbe4f9d9c54 (at 10.0.2.234@o2ib5) Oct 04 10:03:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 10:03:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 04 10:03:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 04 10:06:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 04 10:06:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 04 10:07:09 oak-md1-s2 kernel: LustreError: 11-0: oak-OST001d-osc-MDT0000: operation ost_statfs to node 10.0.2.101@o2ib5 failed: rc = -107 Oct 04 10:07:09 oak-md1-s2 kernel: LustreError: Skipped 252 previous similar messages Oct 04 10:07:09 oak-md1-s2 kernel: Lustre: oak-OST0015-osc-MDT0000: Connection to oak-OST0015 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 04 10:07:09 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Oct 04 10:07:09 oak-md1-s2 kernel: Lustre: oak-OST0023-osc-MDT0000: Connection to oak-OST0023 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 04 10:07:09 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 04 10:07:10 oak-md1-s2 kernel: Lustre: oak-OST0003-osc-MDT0000: Connection to oak-OST0003 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 04 10:07:13 oak-md1-s2 kernel: Lustre: oak-OST0005-osc-MDT0000: Connection to oak-OST0005 (at 10.0.2.101@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 04 10:07:13 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Oct 04 10:07:40 oak-md1-s2 kernel: LNetError: 361386:0:(o2iblnd_cb.c:2759:kiblnd_rejected()) 10.0.2.102@o2ib5 rejected: o2iblnd fatal error Oct 04 10:07:40 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1538672855/real 1538672860] req@ffff8820063f6300 x1610630661300432/t0(0) o8->oak-OST0023-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538672861 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 04 10:07:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7a13ec5f-4b3b-383d-6786-6e860ca9c88a (at 10.9.106.12@o2ib4) reconnecting Oct 04 10:07:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5796daa5-2345-f383-9c22-e0906ac175be (at 10.9.106.12@o2ib4) Oct 04 10:07:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f8f3acec-a37e-6c61-383b-3e5830661ee7 (at 10.9.106.65@o2ib4) reconnecting Oct 04 10:08:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f4f12df2-098d-ac99-28a4-6a8faff710f3 (at 10.9.104.21@o2ib4) Oct 04 10:08:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 10:08:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fa1d9318-3614-d909-899b-49b7c98fb443 (at 10.8.2.2@o2ib6) reconnecting Oct 04 10:08:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 10:08:06 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538672880/real 0] req@ffff8820063f0000 x1610630661306704/t0(0) o8->oak-OST002f-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538672886 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 04 10:08:06 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 10:08:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 284afda2-450f-fd57-ea68-b853668aba28 (at 10.8.3.32@o2ib6) reconnecting Oct 04 10:08:21 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 04 10:08:36 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538672905/real 0] req@ffff8820063f2d00 x1610630661312576/t0(0) o8->oak-OST0023-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1538672916 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 04 10:08:36 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Oct 04 10:08:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 54358924-757b-0312-0e81-72ddaebc2dad (at 10.9.106.26@o2ib4) Oct 04 10:08:38 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Oct 04 10:08:39 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 8 seconds Oct 04 10:09:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 46c13be8-c81e-218c-0a6b-fff8a2938c25 (at 10.8.4.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b275ad800, cur 1538672992 expire 1538672842 last 1538672765 Oct 04 10:10:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b067ccdd-27cb-ae5c-27d9-df62cb606375 (at 10.8.4.19@o2ib6) Oct 04 10:10:01 oak-md1-s2 kernel: Lustre: Skipped 25 previous similar messages Oct 04 10:18:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a7afbacb-688a-2bc2-7304-4d6d92512e4a (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7b11400, cur 1538673502 expire 1538673352 last 1538673275 Oct 04 10:19:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 10:19:37 oak-md1-s2 kernel: Lustre: Skipped 24 previous similar messages Oct 04 10:23:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c2d2fe60-ec52-ddd0-3f7a-ba9481e8f3c1 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061e800400, cur 1538673804 expire 1538673654 last 1538673577 Oct 04 10:26:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 10:32:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c4b6706-43ea-868a-510d-0d130208061d (at 10.0.2.234@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817537d8c00, cur 1538674321 expire 1538674171 last 1538674094 Oct 04 10:52:52 oak-md1-s2 kernel: Lustre: 10513:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538675565/real 1538675565] req@ffff8801f6fc6f00 x1610630662291968/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538675572 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 10:52:52 oak-md1-s2 kernel: Lustre: 10513:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 10:52:59 oak-md1-s2 kernel: Lustre: 10513:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538675572/real 1538675572] req@ffff8801f6fc6f00 x1610630662291968/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538675579 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 10:53:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client caec0415-1c16-81e4-8041-d7a44d2885f6 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db070b800, cur 1538675580 expire 1538675430 last 1538675353 Oct 04 10:53:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 10:58:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 214f6e7f-ae81-121f-a9ad-2fbe4f9d9c54 (at 10.0.2.234@o2ib5) Oct 04 11:00:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d5d70b0-bf87-3164-94c5-f91b3a517b5e (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e1400, cur 1538676030 expire 1538675880 last 1538675803 Oct 04 11:03:57 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 11:04:14 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 11:04:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 11:10:37 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676630/real 1538676630] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676637 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 11:10:44 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676637/real 1538676637] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676644 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:10:51 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676644/real 1538676644] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676651 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:10:58 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676651/real 1538676651] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676658 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:11:05 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676658/real 1538676658] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676665 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:11:19 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676672/real 1538676672] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676679 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:11:19 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 11:11:41 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676693/real 1538676693] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676700 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:11:41 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 11:12:23 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538676736/real 1538676736] req@ffff881894314b00 x1610630662764880/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538676743 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:12:23 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 11:13:05 oak-md1-s2 kernel: LustreError: 10617:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff881894314b00 x1610630662764880 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881cf76e5600/0x6c00c3fb82a1cf60 lrc: 4/0,0 mode: PR/PR res: [0x2000137d5:0x10e68:0x0].0x0 bits 0x13 rrc: 16 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0x30399952c78010f8 expref: 77 pid: 10620 timeout: 6961121170 lvb_type: 0 Oct 04 11:13:05 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 04 11:16:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 11:31:12 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538677865/real 1538677865] req@ffff8800202c1e00 x1610630663269120/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538677872 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 11:31:12 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Oct 04 11:31:26 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538677879/real 1538677879] req@ffff8800202c1e00 x1610630663269120/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538677886 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:31:26 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 11:31:47 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538677900/real 1538677900] req@ffff8800202c1e00 x1610630663269120/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538677907 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:31:47 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 11:32:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ec47e1c-658f-3acf-16b1-8466621cf3a0 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88050764c800, cur 1538677933 expire 1538677783 last 1538677706 Oct 04 11:33:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 11:36:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f930ca3-e8b2-6911-5807-f30582b87d9f (at 10.0.2.234@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880995a93400, cur 1538678209 expire 1538678059 last 1538677982 Oct 04 11:44:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 04 11:44:22 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Oct 04 11:44:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 04 11:45:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d89b266-6824-34bb-8368-b1dd064b4c7b (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daabe4c00, cur 1538678706 expire 1538678556 last 1538678479 Oct 04 11:45:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1d43a15b-9f2d-f47a-e3cf-a596080afa83 (at 10.0.2.3@o2ib5) reconnecting Oct 04 11:53:45 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538679218/real 1538679218] req@ffff880046086900 x1610630667738320/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1538679225 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 11:53:45 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 11:53:52 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538679225/real 1538679225] req@ffff880046086900 x1610630667738320/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1538679232 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:54:07 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538679239/real 1538679239] req@ffff880046086900 x1610630667738320/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1538679246 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:54:07 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 11:54:28 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1538679261/real 1538679254] req@ffff880046086900 x1610630667738320/t0(0) o104->oak-MDT0000@10.0.2.247@o2ib5:15/16 lens 296/224 e 0 to 1 dl 1538679268 ref 2 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 11:54:28 oak-md1-s2 kernel: Lustre: 10561:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 11:55:13 oak-md1-s2 kernel: LNet: 361386:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.247@o2ib5 version 12/12 incarnation 1538093705278552/1538679311846370 Oct 04 11:55:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 357c9327-5943-7b6b-bf71-96f316a93e47 (at 10.0.2.247@o2ib5) Oct 04 11:55:13 oak-md1-s2 kernel: LustreError: 10561:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.0.2.247@o2ib5) returned error from blocking AST (req@ffff880046086900 x1610630667738320 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff88175ce03200/0x6c00c3fb8cd35712 lrc: 4/0,0 mode: PR/PR res: [0x2000167d5:0x4702:0x0].0x0 bits 0x13 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.0.2.247@o2ib5 remote: 0xd1a051a7f8b887b0 expref: 41139 pid: 10631 timeout: 6964406045 lvb_type: 0 Oct 04 11:55:13 oak-md1-s2 kernel: LustreError: 10561:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 1 previous similar message Oct 04 11:55:13 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.0.2.247@o2ib5 was evicted due to a lock blocking callback time out: rc -107 Oct 04 11:55:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 12:10:05 oak-md1-s2 kernel: LNet: 361386:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.243@o2ib5 version 12/12 incarnation 1536020364361230/1538680203813494 Oct 04 12:10:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b86cd386-a2f2-b44a-c0eb-72da3b280f32 (at 10.0.2.243@o2ib5) Oct 04 12:10:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 12:12:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 864ae106-d30f-369b-69e9-dd2fa4c286a9 (at 10.0.2.243@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200be49000, cur 1538680339 expire 1538680189 last 1538680112 Oct 04 12:19:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 66920b67-de3e-3be3-6c64-79054633a5b7 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102dba9c00, cur 1538680742 expire 1538680592 last 1538680515 Oct 04 12:24:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9829246-76ba-6fb0-fc7b-65e149f21c56 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808cdbdd400, cur 1538681066 expire 1538680916 last 1538680839 Oct 04 12:24:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 12:24:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 12:24:37 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 12:31:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 675608a4-837c-230c-ded5-ee0af50a55f5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803d1774400, cur 1538681474 expire 1538681324 last 1538681247 Oct 04 12:40:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 066ba6e1-38f0-3a92-a74e-3936273093d9 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fecf11c00, cur 1538682055 expire 1538681905 last 1538681828 Oct 04 12:42:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 12:42:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 12:48:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 69c8e4b6-a0a4-ba96-fcb5-b307068e40d5 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881812763c00, cur 1538682509 expire 1538682359 last 1538682282 Oct 04 12:54:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 250f31b6-9bb3-924f-175f-2ed23fabac89 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacc58800, cur 1538682841 expire 1538682691 last 1538682614 Oct 04 12:54:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 12:54:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 12:58:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 926eb20e-2410-a82a-73f4-6f593fce6d01 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881befb85000, cur 1538683094 expire 1538682944 last 1538682867 Oct 04 13:20:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02af46f2-96e2-0bde-72b1-55e1458feb71 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000370bc00, cur 1538684440 expire 1538684290 last 1538684213 Oct 04 13:21:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 13:21:58 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 13:29:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b510aa91-eb8f-441c-7b79-2f94e011ed77 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ce878f400, cur 1538684995 expire 1538684845 last 1538684768 Oct 04 13:29:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 13:33:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 13:37:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 47cf49c5-0515-7f67-da5c-d23e377ac991 (at 10.9.0.81@o2ib4) Oct 04 13:44:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09a9e5e5-dc43-8a42-ced0-ab71a4cb5e18 (at 10.12.4.39@o2ib) Oct 04 13:45:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ac4e76db-e781-9abc-1121-0c7ffc05276e (at 10.12.4.54@o2ib) Oct 04 13:45:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 04 13:49:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b2e21f8-1922-74ab-dec7-a4e0da7ed051 (at 10.12.4.49@o2ib) Oct 04 13:49:02 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 13:51:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 070f3e1d-a4dc-b881-9ff3-43a7bbcb9eef (at 10.8.0.82@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186c671400, cur 1538686268 expire 1538686118 last 1538686041 Oct 04 13:58:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e8a5ef39-7928-fd72-8756-b8b6fc5f5e85 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881002c9ac00, cur 1538686695 expire 1538686545 last 1538686468 Oct 04 14:04:05 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:05:01 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:07:34 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:07:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d2ba1d97-8e59-8e00-25a7-a47497099fa2 (at 10.9.104.49@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ee802800, cur 1538687259 expire 1538687109 last 1538687032 Oct 04 14:07:43 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:09:08 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:09:27 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:10:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d25485f4-9864-2174-75c0-eff678521eca (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf206800, cur 1538687417 expire 1538687267 last 1538687190 Oct 04 14:11:48 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:12:26 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 04 14:27:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e8f82620-e480-b4c7-f7a4-247bcbefc4a2 (at 10.8.0.68@o2ib6) Oct 04 14:29:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 04 14:29:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a0ca562-339c-40da-5bda-ff500807d756 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817c2f1ec00, cur 1538688575 expire 1538688425 last 1538688348 Oct 04 14:30:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bba14768-336e-292a-9905-5121b3694a55 (at 10.8.0.67@o2ib6) Oct 04 14:33:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 04 14:33:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 14:34:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be1a6a72-783c-f033-6580-95eea1948534 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d252e9400, cur 1538688880 expire 1538688730 last 1538688653 Oct 04 14:41:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eaa8551e-6b46-fd19-5cdc-872d25721a38 (at 10.9.107.13@o2ib4) Oct 04 14:41:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 14:42:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 000e56bf-e1f6-78ae-6936-b3b42aaf03c4 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800496b1400, cur 1538689328 expire 1538689178 last 1538689101 Oct 04 14:46:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b32d3a08-50ef-4d28-8c61-52f38c0ed599 (at 10.8.21.32@o2ib6) Oct 04 14:46:47 oak-md1-s2 kernel: Lustre: Skipped 30 previous similar messages Oct 04 14:50:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5412003-2660-2ccd-f6a1-82a06c78add1 (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003a19fc00, cur 1538689809 expire 1538689659 last 1538689582 Oct 04 14:56:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 14:56:49 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 04 15:01:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42512709-db37-0d6e-9c11-d60e9afa4a77 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816d4e04c00, cur 1538690462 expire 1538690312 last 1538690235 Oct 04 15:01:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 15:02:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 080f7712-274b-e33c-4129-cf78392bee71 (at 10.12.4.71@o2ib) in 176 seconds. I think it's dead, and I am evicting it. exp ffff8805d4fcf800, cur 1538690538 expire 1538690388 last 1538690362 Oct 04 15:12:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3eac4d27-6938-5bd3-6cf0-77bbdf595f11 (at 10.9.0.61@o2ib4) Oct 04 15:28:54 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538692127/real 1538692127] req@ffff880fc92f5100 x1610630676303168/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538692134 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 15:28:54 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 15:29:01 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538692134/real 1538692134] req@ffff880fc92f5100 x1610630676303168/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538692141 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 15:29:15 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538692148/real 1538692148] req@ffff880fc92f5100 x1610630676303168/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538692155 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 15:29:15 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 15:29:37 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538692169/real 1538692169] req@ffff880fc92f5100 x1610630676303168/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538692176 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 15:29:37 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 15:30:19 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538692212/real 1538692212] req@ffff880fc92f5100 x1610630676303168/t0(0) o104->oak-MDT0000@10.8.4.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538692219 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 15:30:19 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 15:31:22 oak-md1-s2 kernel: LustreError: 10525:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.15@o2ib6) failed to reply to blocking AST (req@ffff880fc92f5100 x1610630676303168 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881c84b01200/0x6c00c3fc09741f92 lrc: 4/0,0 mode: PR/PR res: [0x200013a9e:0x1dd6c:0x0].0x0 bits 0x13 rrc: 58 type: IBT flags: 0x60200400000020 nid: 10.8.4.15@o2ib6 remote: 0xe12d7a4e0fcc929e expref: 125 pid: 10645 timeout: 6976618215 lvb_type: 0 Oct 04 15:31:22 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.15@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 04 15:31:22 oak-md1-s2 kernel: LustreError: Skipped 1 previous similar message Oct 04 15:33:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 15:33:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 15:44:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea44060a-1c03-19c6-4cf0-dce9aa5a445f (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef28d9800, cur 1538693045 expire 1538692895 last 1538692818 Oct 04 15:44:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 15:47:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 16:01:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a8712367-24b3-8a6f-6579-c970f5cb935b (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d5171f000, cur 1538694107 expire 1538693957 last 1538693880 Oct 04 16:05:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 16:11:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b0bd7cc6-a52b-4a83-7857-441acd0aeeb7 (at 10.8.18.16@o2ib6) Oct 04 16:47:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bec5816-5727-7c87-3bb5-16619d3381ff (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565cd000, cur 1538696854 expire 1538696704 last 1538696627 Oct 04 16:59:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 04 17:01:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74e2dd38-10d2-1fb3-1af2-0b39170b2314 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c871800, cur 1538697696 expire 1538697546 last 1538697469 Oct 04 17:06:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 17:12:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb7ef993-fbde-1873-8940-3926fd002463 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b66a22c00, cur 1538698347 expire 1538698197 last 1538698120 Oct 04 17:29:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 04 17:38:12 oak-md1-s2 kernel: Lustre: 10657:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538699885/real 1538699885] req@ffff8814b0260f00 x1610630690074176/t0(0) o104->oak-MDT0000@10.9.105.37@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538699892 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 17:38:12 oak-md1-s2 kernel: Lustre: 10657:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Oct 04 17:38:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 04 17:38:26 oak-md1-s2 kernel: Lustre: 10657:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538699899/real 1538699899] req@ffff8814b0260f00 x1610630690074176/t0(0) o104->oak-MDT0000@10.9.105.37@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538699906 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 17:38:26 oak-md1-s2 kernel: Lustre: 10657:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 04 17:38:47 oak-md1-s2 kernel: Lustre: 10657:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538699920/real 1538699920] req@ffff8814b0260f00 x1610630690074176/t0(0) o104->oak-MDT0000@10.9.105.37@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538699927 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 17:38:47 oak-md1-s2 kernel: Lustre: 10657:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Oct 04 17:39:27 oak-md1-s2 kernel: Lustre: 10548:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538699960/real 1538699960] req@ffff880f4e301200 x1610630690111536/t0(0) o104->oak-MDT0000@10.9.102.22@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538699967 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 17:39:27 oak-md1-s2 kernel: Lustre: 10548:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Oct 04 17:39:51 oak-md1-s2 kernel: LustreError: 10657:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.37@o2ib4) failed to reply to blocking AST (req@ffff8814b0260f00 x1610630690074176 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880d80dc6800/0x6c00c3fc346b2687 lrc: 4/0,0 mode: PR/PR res: [0x2000179db:0x181ec:0x0].0x0 bits 0x1b rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.9.105.37@o2ib4 remote: 0xb87d5c6f2c6f7707 expref: 57575 pid: 10482 timeout: 6984276847 lvb_type: 0 Oct 04 17:39:51 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.37@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Oct 04 17:40:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e9cd6f0-b51f-4e21-818c-b26592c41e6d (at 10.9.105.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f56b10c00, cur 1538700006 expire 1538699856 last 1538699779 Oct 04 17:40:43 oak-md1-s2 kernel: Lustre: 7070:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538700036/real 1538700036] req@ffff88101bea9800 x1610630690164384/t0(0) o104->oak-MDT0000@10.9.105.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538700043 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 17:40:43 oak-md1-s2 kernel: Lustre: 7070:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 62 previous similar messages Oct 04 17:41:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b352f887-16c0-a3fe-0024-421bb4cfec1d (at 10.9.102.16@o2ib4) in 202 seconds. I think it's dead, and I am evicting it. exp ffff881843ed5000, cur 1538700082 expire 1538699932 last 1538699880 Oct 04 17:41:22 oak-md1-s2 kernel: Lustre: Skipped 112 previous similar messages Oct 04 17:41:22 oak-md1-s2 kernel: LustreError: 10526:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880915af3f00 x1610630690187584/t0(0) o104->oak-MDT0000@10.9.102.14@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 04 17:44:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7591cb58-9f67-5f02-5a9c-2131e799fb7a (at 10.9.102.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca4bc00, cur 1538700284 expire 1538700134 last 1538700057 Oct 04 17:44:44 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 04 17:48:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 04 17:57:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c2205683-0c30-1b2d-149a-2ac48caff36c (at 10.8.26.11@o2ib6) Oct 04 17:58:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Oct 04 17:58:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7b1d42fe-33d6-90fe-73b3-710e58c0634b (at 10.8.28.10@o2ib6) Oct 04 17:59:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 11a49266-3eb5-606a-829b-e9916becba6b (at 10.8.23.23@o2ib6) Oct 04 17:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e31134b5-e472-9719-70da-4bce05f885b0 (at 10.8.14.6@o2ib6) Oct 04 17:59:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7737b6c5-b9fd-ad2f-3fc6-e52ae308bc51 (at 10.8.14.5@o2ib6) Oct 04 17:59:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 18:01:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 446824de-5544-c7bd-dc1a-bfc8a593e187 (at 10.9.105.9@o2ib4) Oct 04 18:01:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 18:02:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3a0de41e-5c0b-014e-f10d-b10c9ee82ef9 (at 10.9.104.11@o2ib4) Oct 04 18:02:50 oak-md1-s2 kernel: Lustre: Skipped 14 previous similar messages Oct 04 18:04:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b86e84e-4d6d-685d-67d0-d133d2ce7166 (at 10.9.105.62@o2ib4) Oct 04 18:04:59 oak-md1-s2 kernel: Lustre: Skipped 84 previous similar messages Oct 04 18:07:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 84a57c61-1d45-95da-e4ef-e817d9263056 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fceac5000, cur 1538701624 expire 1538701474 last 1538701397 Oct 04 18:07:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 18:12:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7fc334e9-5c0a-31c2-d2d4-31a9bcf4a4cd (at 10.9.105.71@o2ib4) Oct 04 18:12:01 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Oct 04 18:20:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8fa142bd-b718-6e09-f347-8dbbd9e142a8 (at 10.9.105.70@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d9efac400, cur 1538702419 expire 1538702269 last 1538702192 Oct 04 18:25:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 261abf76-bde0-1ba6-ec7e-c533c252faf4 (at 10.9.104.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b76fbac00, cur 1538702736 expire 1538702586 last 1538702509 Oct 04 18:25:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 18:34:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 00ef3815-e5be-9235-5e7c-fa912eee4bf0 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880004433000, cur 1538703294 expire 1538703144 last 1538703067 Oct 04 18:42:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ccf6ee49-11b8-d2ef-d80a-33dca202c4e1 (at 10.9.105.70@o2ib4) Oct 04 18:42:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 18:45:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6426b699-90b3-e761-7a2f-a2549d206d03 (at 10.9.104.53@o2ib4) Oct 04 18:50:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e6ee2668-6fcf-cd58-82b9-336c3390adc7 (at 10.9.104.72@o2ib4) Oct 04 18:50:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 199a9a2c-7a64-2683-f9ae-60c1692dbf42 (at 10.9.104.70@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88903000, cur 1538704248 expire 1538704098 last 1538704021 Oct 04 18:52:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 82ead652-fe59-05e4-9e34-4fcc5b346469 (at 10.8.18.35@o2ib6) in 210 seconds. I think it's dead, and I am evicting it. exp ffff88103abb5000, cur 1538704324 expire 1538704174 last 1538704114 Oct 04 18:55:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f57690ca-e280-65ed-64c0-1e625d4567a2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c5bb8ec00, cur 1538704520 expire 1538704370 last 1538704293 Oct 04 18:56:54 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538704607/real 1538704607] req@ffff8802a9fbda00 x1610630692540112/t0(0) o104->oak-MDT0000@10.9.113.7@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538704614 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 18:56:54 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Oct 04 18:57:15 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538704628/real 1538704628] req@ffff8802a9fbda00 x1610630692540112/t0(0) o104->oak-MDT0000@10.9.113.7@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538704635 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 18:57:15 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 18:57:57 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538704670/real 1538704670] req@ffff8802a9fbda00 x1610630692540112/t0(0) o104->oak-MDT0000@10.9.113.7@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538704677 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 18:57:57 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 18:57:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 04 18:57:58 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 04 18:59:12 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538704745/real 1538704745] req@ffff88005e473300 x1610630692594720/t0(0) o104->oak-MDT0000@10.8.28.8@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538704752 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 18:59:12 oak-md1-s2 kernel: Lustre: 10605:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Oct 04 18:59:21 oak-md1-s2 kernel: LustreError: 10521:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.113.7@o2ib4) failed to reply to blocking AST (req@ffff8802a9fbda00 x1610630692540112 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880b38ef9600/0x6c00c3fc62adacb8 lrc: 4/0,0 mode: PR/PR res: [0x20001680d:0x1c213:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.9.113.7@o2ib4 remote: 0x6aef6490b4b30735 expref: 27 pid: 10552 timeout: 6989097677 lvb_type: 0 Oct 04 18:59:21 oak-md1-s2 kernel: LustreError: 10521:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 1 previous similar message Oct 04 18:59:21 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.113.7@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Oct 04 18:59:21 oak-md1-s2 kernel: LustreError: Skipped 1 previous similar message Oct 04 18:59:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4cb05941-15cf-3005-cfe4-e42e03ee474f (at 10.8.16.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c64b800, cur 1538704797 expire 1538704647 last 1538704570 Oct 04 19:01:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c17288e-ac69-820b-d2dd-9148ed30a32c (at 10.8.18.35@o2ib6) in 173 seconds. I think it's dead, and I am evicting it. exp ffff880ab2f41c00, cur 1538704873 expire 1538704723 last 1538704700 Oct 04 19:01:13 oak-md1-s2 kernel: Lustre: Skipped 34 previous similar messages Oct 04 19:05:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ab733f40-6ec4-f556-aa64-494535f2e40b (at 10.9.105.40@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184eb19000, cur 1538705110 expire 1538704960 last 1538704883 Oct 04 19:12:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93dd37a0-a6fa-13e3-0cce-818fde1b3658 (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ef8782000, cur 1538705560 expire 1538705410 last 1538705333 Oct 04 19:13:49 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538705622/real 1538705622] req@ffff880071631200 x1610630693008992/t0(0) o104->oak-MDT0000@10.9.105.72@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538705629 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 19:13:49 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Oct 04 19:14:10 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538705643/real 1538705643] req@ffff880071631200 x1610630693008992/t0(0) o104->oak-MDT0000@10.9.105.72@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538705650 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 19:14:10 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 19:14:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dfeefdbc-6862-2573-89f5-8d7aa2631e64 (at 10.9.104.70@o2ib4) Oct 04 19:14:14 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 04 19:14:52 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538705685/real 1538705685] req@ffff880071631200 x1610630693008992/t0(0) o104->oak-MDT0000@10.9.105.72@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538705692 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 19:14:52 oak-md1-s2 kernel: Lustre: 10525:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 19:15:27 oak-md1-s2 kernel: LustreError: 10525:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.72@o2ib4) failed to reply to blocking AST (req@ffff880071631200 x1610630693008992 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8800412e8000/0x6c00c3fc66c9e82d lrc: 4/0,0 mode: PR/PR res: [0x200017a79:0xce4:0x0].0x0 bits 0x13 rrc: 6 type: IBT flags: 0x60200400000020 nid: 10.9.105.72@o2ib4 remote: 0x5ee21a8cd276f4db expref: 136 pid: 10604 timeout: 6990013448 lvb_type: 0 Oct 04 19:15:27 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.72@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Oct 04 19:24:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1b79d1e8-945a-b9cf-fb85-e5f9e3598a24 (at 10.8.2.28@o2ib6) Oct 04 19:24:31 oak-md1-s2 kernel: Lustre: Skipped 25 previous similar messages Oct 04 19:28:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6486e908-a25b-3137-bcdd-3f61c32251c4 (at 10.8.31.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee604e000, cur 1538706520 expire 1538706370 last 1538706293 Oct 04 19:31:53 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538706706/real 1538706706] req@ffff881488a53f00 x1610630693491728/t0(0) o104->oak-MDT0000@10.8.2.20@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538706713 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 19:31:53 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 19:32:07 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538706720/real 1538706720] req@ffff881488a53f00 x1610630693491728/t0(0) o104->oak-MDT0000@10.8.2.20@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538706727 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 19:32:07 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 04 19:32:28 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538706741/real 1538706741] req@ffff881488a53f00 x1610630693491728/t0(0) o104->oak-MDT0000@10.8.2.20@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538706748 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 19:32:28 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 19:32:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7183060-1cc4-ae00-db08-e1603253c626 (at 10.8.2.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bafc3c00, cur 1538706750 expire 1538706600 last 1538706523 Oct 04 19:33:10 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538706783/real 1538706783] req@ffff881488a53f00 x1610630693491728/t0(0) o104->oak-MDT0000@10.8.2.20@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538706790 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 19:33:10 oak-md1-s2 kernel: Lustre: 3871:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 19:33:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e00923e0-1f62-0ae1-76be-1e052d5e78f1 (at 10.8.2.20@o2ib6) in 172 seconds. I think it's dead, and I am evicting it. exp ffff881daa900800, cur 1538706826 expire 1538706676 last 1538706654 Oct 04 19:38:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 063c5f1d-526f-ca26-1ccb-95381185e608 (at 10.9.105.72@o2ib4) Oct 04 19:38:06 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 04 19:43:43 oak-md1-s2 kernel: Lustre: 10524:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538707416/real 1538707416] req@ffff88026e31d700 x1610630693797824/t0(0) o104->oak-MDT0000@10.9.114.8@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538707423 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 19:43:43 oak-md1-s2 kernel: Lustre: 10524:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 19:44:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0e3c43e-b1b0-ae59-986a-734bcffe0a6a (at 10.9.106.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8cac00, cur 1538707488 expire 1538707338 last 1538707261 Oct 04 19:44:48 oak-md1-s2 kernel: LustreError: 4302:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881c3cfd0000 x1610630693832368/t0(0) o104->oak-MDT0000@10.8.17.5@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 04 19:44:48 oak-md1-s2 kernel: LustreError: 4302:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Oct 04 19:48:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e14e759-69f6-ce5d-bdfb-1388f237d02d (at 10.8.18.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880038e8a800, cur 1538707718 expire 1538707568 last 1538707491 Oct 04 19:48:38 oak-md1-s2 kernel: Lustre: Skipped 54 previous similar messages Oct 04 19:52:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b44a08c3-35e8-013e-08a1-a9ea988a8e25 (at 10.8.21.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c870400, cur 1538707930 expire 1538707780 last 1538707703 Oct 04 19:57:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 89dcbd5a-3123-507f-905f-672b2d63dcfd (at 10.8.2.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004b636c00, cur 1538708246 expire 1538708096 last 1538708019 Oct 04 19:57:26 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 04 19:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8a737cd1-8357-eca0-2471-04359d20cdac (at 10.8.2.18@o2ib6) Oct 04 19:59:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 20:01:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0203f52-2043-35e8-c5d8-c9a7438c3e7b (at 10.8.18.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014c053c00, cur 1538708466 expire 1538708316 last 1538708239 Oct 04 20:01:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 20:02:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 00cdaa44-b2c3-f253-a6ea-8d54b5afd9af (at 10.8.1.28@o2ib6) in 212 seconds. I think it's dead, and I am evicting it. exp ffff881bef20e400, cur 1538708542 expire 1538708392 last 1538708330 Oct 04 20:02:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 20:03:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da1b6f98-8420-7580-88a2-105e91d1ccc9 (at 10.8.16.5@o2ib6) in 157 seconds. I think it's dead, and I am evicting it. exp ffff8809439d9800, cur 1538708618 expire 1538708468 last 1538708461 Oct 04 20:04:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3c664464-6c42-82ca-7b9b-46016fbeb290 (at 10.8.17.1@o2ib6) Oct 04 20:04:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 20:07:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6d4765fe-3f90-54e3-5726-62369fd2bc50 (at 10.8.19.5@o2ib6) Oct 04 20:07:13 oak-md1-s2 kernel: Lustre: Skipped 19 previous similar messages Oct 04 20:10:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d383755b-d736-f3d7-b95a-b96741f09b67 (at 10.8.18.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee6048800, cur 1538709023 expire 1538708873 last 1538708796 Oct 04 20:12:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f72c4ffd-cb8b-85d4-fc78-f2d5a62e22c4 (at 10.8.8.2@o2ib6) Oct 04 20:12:14 oak-md1-s2 kernel: Lustre: Skipped 33 previous similar messages Oct 04 20:17:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9279300a-e56d-3e06-264b-91fc3ba9375a (at 10.8.18.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee604d000, cur 1538709469 expire 1538709319 last 1538709242 Oct 04 20:17:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 20:23:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 11381945-20c9-61f0-9b6f-ad7bf7cf174f (at 10.8.16.5@o2ib6) Oct 04 20:23:16 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Oct 04 20:27:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 89db9227-9060-44b4-1114-9adf7ab456e8 (at 10.8.16.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88170039e800, cur 1538710072 expire 1538709922 last 1538709845 Oct 04 20:27:52 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 04 20:35:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 93746428-2f58-7d78-871d-b7b2cb66dc27 (at 10.8.18.20@o2ib6) Oct 04 20:35:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 04 20:38:46 oak-md1-s2 kernel: Lustre: 10540:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538710719/real 1538710719] req@ffff881488a56600 x1610630696173904/t0(0) o104->oak-MDT0000@10.9.102.30@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538710726 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 04 20:38:46 oak-md1-s2 kernel: Lustre: 10540:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Oct 04 20:39:07 oak-md1-s2 kernel: Lustre: 10540:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538710740/real 1538710740] req@ffff881488a56600 x1610630696173904/t0(0) o104->oak-MDT0000@10.9.102.30@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538710747 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 20:39:07 oak-md1-s2 kernel: Lustre: 10540:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 04 20:39:49 oak-md1-s2 kernel: Lustre: 10540:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538710782/real 1538710782] req@ffff881488a56600 x1610630696173904/t0(0) o104->oak-MDT0000@10.9.102.30@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538710789 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 04 20:39:49 oak-md1-s2 kernel: Lustre: 10540:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 04 20:40:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 03d617ee-b886-60e9-bca1-2035cf92278f (at 10.9.102.30@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185caa0000, cur 1538710858 expire 1538710708 last 1538710631 Oct 04 20:40:58 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 04 20:53:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c32ea898-d706-20c1-e2ae-e44cd16f96da (at 10.8.29.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b29bfd000, cur 1538711597 expire 1538711447 last 1538711370 Oct 04 20:53:17 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 04 20:53:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to abf9b470-5bf2-0d56-18d9-117f8957fa53 (at 10.8.21.36@o2ib6) Oct 04 20:53:55 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 04 21:03:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 86aff0d8-19be-42c3-dbf6-625ce9434bf6 (at 10.9.102.71@o2ib4) Oct 04 21:03:56 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Oct 04 21:09:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7ff2520f-27bd-d58e-5394-414b649c7a50 (at 10.9.102.68@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d18d0d400, cur 1538712581 expire 1538712431 last 1538712354 Oct 04 21:09:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 21:15:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4a686cab-f3c9-0cf1-6b97-ed07e9b8fa40 (at 10.9.102.60@o2ib4) Oct 04 21:15:16 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 04 21:32:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8b826015-ca49-b9c4-2bad-3ad8777ef035 (at 10.9.102.68@o2ib4) Oct 04 21:32:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 22:00:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dec1e6cf-0646-d970-10f0-502161882a3b (at 10.8.13.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88010988b800, cur 1538715621 expire 1538715471 last 1538715394 Oct 04 22:00:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 22:01:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b3bbabc-ae33-a8a0-e0e9-1ef2c52750d0 (at 10.8.19.7@o2ib6) in 225 seconds. I think it's dead, and I am evicting it. exp ffff88185ff53800, cur 1538715697 expire 1538715547 last 1538715472 Oct 04 22:08:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a13afa74-84ee-252e-2467-3f545dc17537 (at 10.8.13.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edfc000, cur 1538716093 expire 1538715943 last 1538715866 Oct 04 22:08:13 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 22:16:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 57d93be5-1603-d8ba-e857-f0b78ec56438 (at 10.8.19.7@o2ib6) Oct 04 22:16:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 22:20:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 761e2109-61ee-02a6-cbe8-81b45db1b50f (at 10.8.13.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3f3800, cur 1538716807 expire 1538716657 last 1538716580 Oct 04 22:20:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 22:20:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b1ab462c-3e89-0093-db3a-69918591ae2a (at 10.8.26.2@o2ib6) Oct 04 22:25:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbdf5d16-42da-afcc-a6a4-e938f9885d13 (at 10.8.13.19@o2ib6) Oct 04 22:25:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 22:31:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 24a11447-53d3-3b5e-871f-dfabe2a1b62a (at 10.9.102.58@o2ib4) Oct 04 22:32:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5d9c2a87-8884-419e-d697-532dc38f5263 (at 10.9.103.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6c8dc000, cur 1538717545 expire 1538717395 last 1538717318 Oct 04 22:43:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 761e2109-61ee-02a6-cbe8-81b45db1b50f (at 10.8.13.1@o2ib6) Oct 04 22:43:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 22:54:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae1b5bfa-1502-d183-f778-ff932572fd0b (at 10.9.105.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880530fdc400, cur 1538718890 expire 1538718740 last 1538718663 Oct 04 22:54:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 22:55:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5d9c2a87-8884-419e-d697-532dc38f5263 (at 10.9.103.6@o2ib4) Oct 04 22:55:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 04 23:09:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ecbab6dc-8920-0efe-3221-46a054e44ecb (at 10.8.21.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c875000, cur 1538719797 expire 1538719647 last 1538719570 Oct 04 23:09:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 04 23:17:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eb667cc3-eb53-5f57-70db-256d018b1564 (at 10.9.105.15@o2ib4) Oct 04 23:19:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6d6325a1-375a-125b-ca17-b428d1f0293f (at 10.9.105.16@o2ib4) Oct 04 23:19:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 04 23:19:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e13a57f8-bd26-72a6-03a0-a372d99e46fa (at 10.8.31.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e19c44c00, cur 1538720384 expire 1538720234 last 1538720157 Oct 04 23:30:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4cc34ad7-6731-a63a-a8ca-29cfd314359d (at 10.8.21.8@o2ib6) Oct 04 23:37:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d056d80d-436e-5c22-cda6-50ef052dee5a (at 10.8.31.3@o2ib6) Oct 05 00:09:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1bbdb943-b832-2927-469f-b0f12e476d20 (at 10.9.103.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801a0e59c00, cur 1538723397 expire 1538723247 last 1538723170 Oct 05 00:19:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9ce75b7-6154-39c6-a0d2-668fbf297bc0 (at 10.8.17.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af2400, cur 1538723957 expire 1538723807 last 1538723730 Oct 05 00:19:43 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538723976/real 1538723976] req@ffff881806e2c500 x1610630720916080/t0(0) o104->oak-MDT0000@10.9.102.31@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538723983 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 05 00:19:43 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Oct 05 00:19:58 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538723990/real 1538723990] req@ffff881806e2c500 x1610630720916080/t0(0) o104->oak-MDT0000@10.9.102.31@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538723997 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 05 00:19:58 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 05 00:20:19 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538724012/real 1538724012] req@ffff881806e2c500 x1610630720916080/t0(0) o104->oak-MDT0000@10.9.102.31@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538724019 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 05 00:20:19 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Oct 05 00:21:01 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538724054/real 1538724054] req@ffff881806e2c500 x1610630720916080/t0(0) o104->oak-MDT0000@10.9.102.31@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1538724061 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 05 00:21:01 oak-md1-s2 kernel: Lustre: 4678:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Oct 05 00:22:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6794604c-2ae9-3461-1d4b-ae726d193e2c (at 10.9.102.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0e6400, cur 1538724124 expire 1538723974 last 1538723897 Oct 05 00:32:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b4a8fa9f-d86d-5610-8203-8036ab157a8a (at 10.8.17.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88048a9ee400, cur 1538724768 expire 1538724618 last 1538724541 Oct 05 00:32:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 00:32:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1bbdb943-b832-2927-469f-b0f12e476d20 (at 10.9.103.20@o2ib4) Oct 05 00:37:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e282a2c1-e19b-54e8-7a33-bbb74a09b2a4 (at 10.8.17.16@o2ib6) Oct 05 00:39:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fb68cf8f-ae40-20d5-2e4e-ae4e7cbfd134 (at 10.8.13.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881eccec9800, cur 1538725195 expire 1538725045 last 1538724968 Oct 05 00:39:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 00:41:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e77b854a-e4df-1376-09e4-06c395944567 (at 10.8.24.2@o2ib6) in 154 seconds. I think it's dead, and I am evicting it. exp ffff880e19c46c00, cur 1538725271 expire 1538725121 last 1538725117 Oct 05 00:45:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e7f7e432-db3d-fd29-36c0-946bc794ceed (at 10.9.102.31@o2ib4) Oct 05 00:51:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b4a8fa9f-d86d-5610-8203-8036ab157a8a (at 10.8.17.18@o2ib6) Oct 05 00:51:18 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 00:57:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dcf7ebae-b4a3-9078-cfb7-d45292c01400 (at 10.9.102.44@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200b3de000, cur 1538726256 expire 1538726106 last 1538726029 Oct 05 01:01:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1b75d504-de7f-1842-1d35-1b833c34cbcb (at 10.8.29.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803352b4000, cur 1538726466 expire 1538726316 last 1538726239 Oct 05 01:03:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 68988b6e-a6a4-5047-9b52-744a2a89b2e1 (at 10.8.13.21@o2ib6) Oct 05 01:03:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 01:04:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb740ae1-e20a-02a7-1cfc-0b71710a8174 (at 10.8.29.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4fc00, cur 1538726685 expire 1538726535 last 1538726458 Oct 05 01:07:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 058b6278-177a-f759-6963-9e13604935c3 (at 10.8.17.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1f400, cur 1538726840 expire 1538726690 last 1538726613 Oct 05 01:10:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 488d9048-5978-7ef0-6f73-ceb8ba95cf1c (at 10.8.29.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810302f0800, cur 1538727016 expire 1538726866 last 1538726789 Oct 05 01:14:36 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 01:14:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3d485529-d246-2b3f-7794-d392406ac0e1 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e6800, cur 1538727293 expire 1538727143 last 1538727066 Oct 05 01:15:13 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 01:15:13 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 05 01:15:34 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 01:15:34 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 05 01:16:15 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 01:16:15 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 24 previous similar messages Oct 05 01:18:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6df9f485-8ac4-4758-2045-ffad4e1b85f1 (at 10.8.29.3@o2ib6) Oct 05 01:31:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Oct 05 01:31:59 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 01:32:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d27591f3-7dfc-1d68-3e81-c1bcbcd823ae (at 10.8.29.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881eccece400, cur 1538728345 expire 1538728195 last 1538728118 Oct 05 01:32:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 01:38:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b52911eb-5c96-fa24-47ea-fdd3f2150447 (at 10.9.102.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b76fb8000, cur 1538728698 expire 1538728548 last 1538728471 Oct 05 01:44:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 10dc127f-730d-4058-0cd4-515fae342d51 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bbaf8400, cur 1538729067 expire 1538728917 last 1538728840 Oct 05 01:44:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 01:47:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e69305dd-c324-76b2-02a0-2ddef3efb42b (at 10.8.29.8@o2ib6) Oct 05 01:47:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 01:58:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d83aca7-e3c9-4f5e-7360-790dc57907f3 (at 10.8.13.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88170039ac00, cur 1538729896 expire 1538729746 last 1538729669 Oct 05 02:01:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 05 02:01:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 02:08:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 808a837b-ee01-ac04-f18d-65a5cde4d0d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d2696800, cur 1538730491 expire 1538730341 last 1538730264 Oct 05 02:21:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7d83aca7-e3c9-4f5e-7360-790dc57907f3 (at 10.8.13.17@o2ib6) Oct 05 02:21:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 02:22:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2aefdbdd-ac3c-6cc8-ba89-9645df2fcc88 (at 10.8.4.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c7fa26800, cur 1538731371 expire 1538731221 last 1538731144 Oct 05 02:37:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 624c529f-e617-57be-caeb-81ba6487a0b9 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d2695400, cur 1538732222 expire 1538732072 last 1538731995 Oct 05 02:40:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a39b5e7-228a-4d42-7b5a-21f2d9d158ad (at 10.8.13.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af2800, cur 1538732458 expire 1538732308 last 1538732231 Oct 05 02:46:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f2542864-5025-7878-8d3e-769f383ab97e (at 10.8.4.36@o2ib6) Oct 05 02:46:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 02:55:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1f95f706-3e1d-f062-0b96-ad1ab136fca6 (at 10.8.23.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef3c00, cur 1538733307 expire 1538733157 last 1538733080 Oct 05 02:55:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 02:57:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38faf909-d73c-405b-5f5d-d1704d815d87 (at 10.8.29.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa014400, cur 1538733472 expire 1538733322 last 1538733245 Oct 05 02:58:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 986bf173-8a48-8326-0b87-c02bc7db3b53 (at 10.8.9.8@o2ib6) Oct 05 03:07:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c2068c10-7c4f-34b2-dd40-6bd812b10ffd (at 10.9.102.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b99266c00, cur 1538734032 expire 1538733882 last 1538733805 Oct 05 03:11:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 436b24aa-d26a-36a5-78b6-ec26a3fda5e5 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa872800, cur 1538734273 expire 1538734123 last 1538734046 Oct 05 03:13:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1f95f706-3e1d-f062-0b96-ad1ab136fca6 (at 10.8.23.2@o2ib6) Oct 05 03:13:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 03:14:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fd8236e-050b-c62d-2664-f6711708a9ca (at 10.8.29.6@o2ib6) Oct 05 03:26:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7546526b-e8c4-b494-8212-dd8e0336268c (at 10.8.29.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103b5f6400, cur 1538735176 expire 1538735026 last 1538734949 Oct 05 03:29:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 156573be-eca5-3a48-3b0f-a12ef5bbd1ab (at 10.9.102.26@o2ib4) Oct 05 03:37:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fd8236e-050b-c62d-2664-f6711708a9ca (at 10.8.29.6@o2ib6) Oct 05 03:48:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7aeb2b0-ca12-8bfc-7e57-adaee8acba63 (at 10.8.2.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bafc4400, cur 1538736523 expire 1538736373 last 1538736296 Oct 05 03:52:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 13ed2aa2-e9ce-44ed-47cd-6abd49f1cabf (at 10.8.6.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fecf03c00, cur 1538736743 expire 1538736593 last 1538736516 Oct 05 04:01:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 492413ca-9289-1fc5-5559-3ea635453bc7 (at 10.9.102.49@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7405400, cur 1538737270 expire 1538737120 last 1538737043 Oct 05 04:13:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 91c0a371-2eb7-6af5-f927-a00df13ee31f (at 10.8.2.6@o2ib6) Oct 05 04:13:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d94b3400-fce6-594b-5736-66311e99e544 (at 10.8.6.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812642aac00, cur 1538738017 expire 1538737867 last 1538737790 Oct 05 04:13:37 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 04:16:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d5bbb2f3-d359-d4c1-038d-52b7d34607f5 (at 10.8.6.2@o2ib6) Oct 05 04:23:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e25b6204-7903-2636-00f7-ddfa5235be5b (at 10.9.102.67@o2ib4) Oct 05 04:24:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bd78d402-192f-02ba-dfdc-71a371ef8b56 (at 10.9.102.59@o2ib4) Oct 05 04:25:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 740168ad-fda3-eb8f-8fa9-f92503b158d4 (at 10.9.102.69@o2ib4) Oct 05 04:25:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 04:34:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b58c9f6-4e56-0c10-eb14-5cb3bbdadac5 (at 10.8.2.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88197aa5c800, cur 1538739247 expire 1538739097 last 1538739020 Oct 05 04:35:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37f8a247-3aec-9a8a-15fd-baaa6bb067c2 (at 10.8.19.6@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff8802a9bb9400, cur 1538739323 expire 1538739173 last 1538739097 Oct 05 04:35:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 04:37:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5bf085c4-7698-cb73-d91c-d388aeb8271a (at 10.8.2.14@o2ib6) Oct 05 04:37:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 04:38:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f52d7fa9-a491-9d23-3523-b1afd1fdf6f0 (at 10.8.6.10@o2ib6) Oct 05 04:40:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c03bf188-4c1a-7f4d-4d3c-e4d37a0dde67 (at 10.8.10.36@o2ib6) Oct 05 04:40:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c1b7fca-b552-77cc-2226-aa83bb645c6d (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ed252800, cur 1538739642 expire 1538739492 last 1538739415 Oct 05 04:48:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da81cd8a-54bb-1bcc-ecce-162331019b10 (at 10.9.103.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e7a4000, cur 1538740120 expire 1538739970 last 1538739893 Oct 05 04:49:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f9119f12-7165-2f34-24c2-b98186c12e77 (at 10.8.19.1@o2ib6) Oct 05 04:50:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ac7b8475-251b-16ac-f682-d7b7d6e192fd (at 10.8.19.6@o2ib6) Oct 05 04:54:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d23d599-90aa-377c-9092-d44bd10ce174 (at 10.8.17.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880109889400, cur 1538740496 expire 1538740346 last 1538740269 Oct 05 05:12:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to da81cd8a-54bb-1bcc-ecce-162331019b10 (at 10.9.103.5@o2ib4) Oct 05 05:13:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 17b6e27b-ecd6-64c5-3e5e-45a542de047a (at 10.8.17.17@o2ib6) Oct 05 06:16:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef8b2bfb-804b-0945-47e9-30d4328a2510 (at 10.8.1.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88197aa5cc00, cur 1538745388 expire 1538745238 last 1538745161 Oct 05 06:40:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6289a368-dd6e-c696-8bba-52642ee3ef0d (at 10.8.1.32@o2ib6) Oct 05 06:40:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6287b08e-7a2b-d21f-9862-c53903ea650f (at 10.8.1.33@o2ib6) Oct 05 06:42:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2e443e22-214e-df90-07fc-c40496ad920c (at 10.8.1.26@o2ib6) Oct 05 06:50:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a85b2c82-84ae-199d-ff6a-fceab5148276 (at 10.8.3.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa904c00, cur 1538747458 expire 1538747308 last 1538747231 Oct 05 06:50:58 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 07:11:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 15a10e83-566c-bbc9-4b76-208d51c74223 (at 10.9.103.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a277800, cur 1538748670 expire 1538748520 last 1538748443 Oct 05 07:12:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 988a063a-1940-2a7a-761f-41d42cd3395d (at 10.9.104.40@o2ib4) in 215 seconds. I think it's dead, and I am evicting it. exp ffff881868a0e400, cur 1538748746 expire 1538748596 last 1538748531 Oct 05 07:12:26 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 07:14:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d036a5bf-7ec8-4c86-6d42-02f20fb9cd2f (at 10.8.3.12@o2ib6) Oct 05 07:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 584d7ee1-229d-ebb8-0c06-c35918513078 (at 10.9.104.39@o2ib4) Oct 05 07:35:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 33342942-928f-5dec-33dc-9ac7dee8cd0b (at 10.9.104.43@o2ib4) Oct 05 07:35:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e3844a92-0aa4-4a01-81b4-7f245fa7414c (at 10.9.104.38@o2ib4) Oct 05 07:35:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 15a10e83-566c-bbc9-4b76-208d51c74223 (at 10.9.103.8@o2ib4) Oct 05 07:36:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e94d6118-23f4-e3f9-21de-ead75e01bb9a (at 10.9.102.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185ff51c00, cur 1538750172 expire 1538750022 last 1538749945 Oct 05 07:36:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 026219c3-9078-ff1b-5145-6d92dbb68dd4 (at 10.9.104.44@o2ib4) Oct 05 07:36:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b685d407-1c0a-9e66-d939-ab0c2e7f5043 (at 10.9.104.40@o2ib4) Oct 05 07:43:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7f68d6e-9892-83a5-296f-7ed402ae9049 (at 10.8.18.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e3800, cur 1538750613 expire 1538750463 last 1538750386 Oct 05 07:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a20e73e0-2fdb-e334-ee18-862b441fbdce (at 10.9.102.48@o2ib4) in 203 seconds. I think it's dead, and I am evicting it. exp ffff880fd8afec00, cur 1538750689 expire 1538750539 last 1538750486 Oct 05 07:44:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 07:48:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b3a82a8-5987-2c98-000c-565957b7fb10 (at 10.9.102.38@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1c0b8c00, cur 1538750936 expire 1538750786 last 1538750709 Oct 05 07:48:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 07:50:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55095e98-8253-7037-02d9-98c0c47ca95f (at 10.9.102.42@o2ib4) in 197 seconds. I think it's dead, and I am evicting it. exp ffff881f927e0400, cur 1538751012 expire 1538750862 last 1538750815 Oct 05 07:50:12 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 07:51:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client afcabddf-ef72-18d7-24b8-64be64e8dafb (at 10.9.103.18@o2ib4) in 221 seconds. I think it's dead, and I am evicting it. exp ffff882005c35000, cur 1538751088 expire 1538750938 last 1538750867 Oct 05 07:51:28 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 07:55:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb93609f-51d1-76e4-02b9-a6dd2112e2c9 (at 10.8.13.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3f1c00, cur 1538751356 expire 1538751206 last 1538751129 Oct 05 07:55:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 07:58:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 79f0c6c7-0b2e-0258-fd29-427810f8977d (at 10.9.102.62@o2ib4) Oct 05 07:58:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 08:02:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e5152a9c-eb06-d6e8-2102-42f90e2de340 (at 10.8.18.1@o2ib6) Oct 05 08:02:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7981e39f-f707-32d0-ce64-44706126f1d0 (at 10.8.18.7@o2ib6) Oct 05 08:07:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b852b871-4461-3539-0078-5bd024b1f0e3 (at 10.9.102.48@o2ib4) Oct 05 08:08:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2c181141-00e9-9dd2-49c2-18dde8e229ef (at 10.9.102.47@o2ib4) Oct 05 08:10:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 946940ec-c71f-9ee7-5c8b-23830a30b395 (at 10.9.102.38@o2ib4) Oct 05 08:12:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ecd5a14-5867-79c1-8dc0-a713c6faf6dd (at 10.9.102.33@o2ib4) Oct 05 08:12:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 08:15:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5e12c999-3b28-724d-a64c-7800792823ff (at 10.9.102.35@o2ib4) Oct 05 08:15:36 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 05 08:20:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bb93609f-51d1-76e4-02b9-a6dd2112e2c9 (at 10.8.13.4@o2ib6) Oct 05 08:20:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 08:21:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f8e8113e-defb-5365-5ffb-c997185fd598 (at 10.8.13.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3f0800, cur 1538752892 expire 1538752742 last 1538752665 Oct 05 08:29:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 05 08:30:39 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538753432/real 1538753432] req@ffff88001b33a700 x1610630739341856/t0(0) o104->oak-MDT0000@10.8.7.21@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538753439 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 05 08:30:39 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Oct 05 08:30:53 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538753446/real 1538753446] req@ffff88001b33a700 x1610630739341856/t0(0) o104->oak-MDT0000@10.8.7.21@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538753453 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 05 08:30:53 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 05 08:31:14 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1538753467/real 1538753467] req@ffff88001b33a700 x1610630739341856/t0(0) o104->oak-MDT0000@10.8.7.21@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1538753474 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 05 08:31:14 oak-md1-s2 kernel: Lustre: 10521:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 05 08:31:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5511f267-857d-a55b-0f00-60542f1c191c (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cc3ef8000, cur 1538753505 expire 1538753355 last 1538753278 Oct 05 08:31:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 08:31:45 oak-md1-s2 kernel: LustreError: 10631:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88004f300000 x1610630739367712/t0(0) o104->oak-MDT0000@10.8.7.21@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 05 08:31:48 oak-md1-s2 kernel: LustreError: 7070:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88001358d100 x1610630739368960/t0(0) o104->oak-MDT0000@10.8.7.21@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 05 08:39:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f3d93ad-a840-8859-7645-a42efff829d0 (at 10.8.24.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185d7d3c00, cur 1538753946 expire 1538753796 last 1538753719 Oct 05 08:39:06 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 05 08:45:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6319d288-4dab-95f4-ec00-986e3dde6b10 (at 10.8.13.18@o2ib6) Oct 05 08:45:12 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 08:51:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ad63a976-8847-ac92-a5f2-7836b1dd430f (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa907400, cur 1538754681 expire 1538754531 last 1538754454 Oct 05 08:54:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f53838b7-cc9a-f575-bf85-9430aa1e6454 (at 10.8.20.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c64c000, cur 1538754883 expire 1538754733 last 1538754656 Oct 05 08:56:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8f3d93ad-a840-8859-7645-a42efff829d0 (at 10.8.24.10@o2ib6) Oct 05 08:56:10 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 09:13:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f53838b7-cc9a-f575-bf85-9430aa1e6454 (at 10.8.20.29@o2ib6) Oct 05 09:13:22 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 09:22:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9d206e7-998a-63f3-4be1-3e582ba6798b (at 10.9.102.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881888aa4400, cur 1538756544 expire 1538756394 last 1538756317 Oct 05 09:22:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 09:27:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ac4f767-4b5e-4fe3-01ac-bbee6ed40f49 (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880551970400, cur 1538756833 expire 1538756683 last 1538756606 Oct 05 09:30:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 05 09:30:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 09:38:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3cd9d78-11a0-3445-044e-143d18276c75 (at 10.8.20.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef1800, cur 1538757519 expire 1538757369 last 1538757292 Oct 05 09:42:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 05 09:45:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client af996111-ba36-7724-c267-0682f3586a42 (at 10.8.13.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef2800, cur 1538757946 expire 1538757796 last 1538757719 Oct 05 09:45:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 09:50:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b6b861a-54ea-087e-1b4c-76ec74e19de0 (at 10.8.26.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa016800, cur 1538758245 expire 1538758095 last 1538758018 Oct 05 09:59:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d3cd9d78-11a0-3445-044e-143d18276c75 (at 10.8.20.21@o2ib6) Oct 05 09:59:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 10:05:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 385534c5-b40e-3c7c-84b2-72afcf05a570 (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809762ec000, cur 1538759150 expire 1538759000 last 1538758923 Oct 05 10:05:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 10:08:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5f137b0-0f5e-0f66-cce0-72e7891f1bc5 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058c465400, cur 1538759304 expire 1538759154 last 1538759077 Oct 05 10:09:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to af996111-ba36-7724-c267-0682f3586a42 (at 10.8.13.22@o2ib6) Oct 05 10:09:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 10:19:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 600217b9-5a35-5619-8322-6d1969c47f30 (at 10.8.8.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a30d800, cur 1538759974 expire 1538759824 last 1538759747 Oct 05 10:19:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 10:20:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9936049-e69c-6088-e01b-f611d94de001 (at 10.8.8.22@o2ib6) in 181 seconds. I think it's dead, and I am evicting it. exp ffff8801530ee000, cur 1538760050 expire 1538759900 last 1538759869 Oct 05 10:24:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4599d6b2-ef9d-7270-54e6-af2dece8a94c (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3f1000, cur 1538760272 expire 1538760122 last 1538760045 Oct 05 10:24:32 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 10:24:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c328a9ac-da17-ec8e-0947-0f682a2ee0ef (at 10.8.26.10@o2ib6) Oct 05 10:40:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 032e647c-1526-9006-16da-fa6128a35d90 (at 10.8.17.24@o2ib6) Oct 05 10:40:52 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 10:44:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac9b1d64-d16d-9653-2cfc-761b66d28c5c (at 10.8.21.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c64d000, cur 1538761447 expire 1538761297 last 1538761220 Oct 05 11:01:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ac9b1d64-d16d-9653-2cfc-761b66d28c5c (at 10.8.21.29@o2ib6) Oct 05 11:01:28 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 05 11:01:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 76618a18-e698-65ad-fd61-f97f71424b39 (at 10.8.23.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d32eebc00, cur 1538762490 expire 1538762340 last 1538762263 Oct 05 11:08:20 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 11:08:20 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Oct 05 11:09:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7cb215d1-0de5-3264-2c21-7953e6271d9b (at 10.8.18.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e44800, cur 1538762968 expire 1538762818 last 1538762741 Oct 05 11:10:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8edfb5f5-3b72-5ea8-038a-9312b716a3d3 (at 10.8.28.5@o2ib6) in 152 seconds. I think it's dead, and I am evicting it. exp ffff88185ff50400, cur 1538763044 expire 1538762894 last 1538762892 Oct 05 11:10:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 11:12:50 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 11:15:38 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 11:16:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ace491b2-df19-dac5-8850-93f12bfcc7f0 (at 10.9.112.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db0212400, cur 1538763407 expire 1538763257 last 1538763180 Oct 05 11:17:12 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 11:17:12 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 05 11:19:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e90ff330-3085-7576-cce2-638790dc16d3 (at 10.8.23.21@o2ib6) Oct 05 11:20:58 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 11:28:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2b32d60e-17c0-3d0b-c4ed-75dd9ae1d0f6 (at 10.8.18.4@o2ib6) Oct 05 11:28:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b8165cde-d5fa-2a3b-7949-1d2dca4dc294 (at 10.8.18.3@o2ib6) Oct 05 11:30:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 280d00f3-39c5-38de-5260-74de973b3f34 (at 10.8.28.5@o2ib6) Oct 05 11:30:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 11:31:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3421e93-fae8-00ca-fb76-98b8844f1714 (at 10.9.112.12@o2ib4) Oct 05 11:39:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e68002cf-86cc-964d-74ea-0c6854b19bfe (at 10.0.2.224@o2ib5) reconnecting Oct 05 11:39:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42ddaac4-4249-5052-00c0-3d0a27940cdb (at 10.0.2.224@o2ib5) Oct 05 11:39:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e68002cf-86cc-964d-74ea-0c6854b19bfe (at 10.0.2.224@o2ib5) reconnecting Oct 05 11:54:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 05 11:54:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 11:56:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a99bbea-ab39-c541-2335-f20af1ba7178 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88006735b000, cur 1538765818 expire 1538765668 last 1538765591 Oct 05 11:59:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f956d5d4-b21a-dc62-344a-b910d16a608d (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e6ba09c00, cur 1538765991 expire 1538765841 last 1538765764 Oct 05 11:59:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 12:03:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8b3afc61-a1c5-d255-49f4-c2af3fa37430 (at 10.9.104.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103b5f5800, cur 1538766193 expire 1538766043 last 1538765966 Oct 05 12:03:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 12:04:21 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 12:04:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dddb98e0-ef75-8127-bbdf-06be9eecc098 (at 10.9.102.72@o2ib4) in 200 seconds. I think it's dead, and I am evicting it. exp ffff880197a55400, cur 1538766269 expire 1538766119 last 1538766069 Oct 05 12:04:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 12:15:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c92f660f-1fc6-2ff1-5f73-8d9dc2a5eabc (at 10.8.18.18@o2ib6) Oct 05 12:25:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 05 12:25:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 40049887-7569-bc95-6b8b-f92d9b6bfd81 (at 10.9.102.29@o2ib4) Oct 05 12:26:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ca9b33c4-4931-6293-f9ef-c9130adc5da7 (at 10.9.105.36@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a5f77a800, cur 1538767597 expire 1538767447 last 1538767370 Oct 05 12:27:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7f5d0314-d095-6394-699f-52fe02328244 (at 10.9.104.46@o2ib4) Oct 05 12:27:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dddb98e0-ef75-8127-bbdf-06be9eecc098 (at 10.9.102.72@o2ib4) Oct 05 12:45:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 448c6006-5779-fb41-f56f-739b6c7b242d (at 10.9.115.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac8fc400, cur 1538768754 expire 1538768604 last 1538768527 Oct 05 12:50:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1572582a-3774-723f-98a5-09f5833e7d04 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f712800, cur 1538769012 expire 1538768862 last 1538768785 Oct 05 12:50:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 415c02a0-7be6-dd07-c9f5-5b30f452d4ee (at 10.9.105.36@o2ib4) Oct 05 12:55:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1f7bb35e-98be-4838-c6b1-e1bfe5a12dad (at 10.8.6.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c7fa25400, cur 1538769353 expire 1538769203 last 1538769126 Oct 05 13:03:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 87665de1-4903-c365-c80a-c8ffe4dc3730 (at 10.9.115.3@o2ib4) Oct 05 13:10:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 00d69dfd-a071-f8e7-0d63-6733e151298c (at 10.8.12.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e43800, cur 1538770234 expire 1538770084 last 1538770007 Oct 05 13:12:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 05 13:14:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0414d083-ef7b-c692-c57c-fd96eaf56eb1 (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882014838400, cur 1538770482 expire 1538770332 last 1538770255 Oct 05 13:20:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9eba8b2e-5027-fef5-214a-03599f79e7df (at 10.9.104.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809dc2a1000, cur 1538770815 expire 1538770665 last 1538770588 Oct 05 13:20:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e2d34a59-0642-6176-93a0-aa057392cde5 (at 10.8.6.29@o2ib6) Oct 05 13:34:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 00d69dfd-a071-f8e7-0d63-6733e151298c (at 10.8.12.29@o2ib6) Oct 05 13:42:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6d62340-b695-e90f-676c-a56d84d60d81 (at 10.9.104.64@o2ib4) Oct 05 13:42:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e054ea7f-9af6-6009-eb9a-5c9eddc957f4 (at 10.9.104.67@o2ib4) Oct 05 13:43:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a4497593-a920-53fa-0eda-ffe87383d571 (at 10.9.106.29@o2ib4) Oct 05 13:44:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e0a2db5a-a04f-2beb-7817-bdc136c7d38c (at 10.9.104.59@o2ib4) Oct 05 13:46:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 162df202-e71a-9455-6e19-b346fb95d45c (at 10.9.106.44@o2ib4) Oct 05 13:46:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 13:46:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e675494-a646-a3b3-596a-07442292338c (at 10.9.101.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185bacf400, cur 1538772410 expire 1538772260 last 1538772183 Oct 05 13:46:50 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 13:48:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 70ef51bb-672e-05c1-02de-c220016382ab (at 10.8.14.1@o2ib6) in 192 seconds. I think it's dead, and I am evicting it. exp ffff880b2f717400, cur 1538772486 expire 1538772336 last 1538772294 Oct 05 13:48:06 oak-md1-s2 kernel: Lustre: Skipped 181 previous similar messages Oct 05 13:53:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 903946f9-01ce-1ec0-39aa-029292d7faf0 (at 10.9.107.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019e612c00, cur 1538772811 expire 1538772661 last 1538772584 Oct 05 14:08:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 211f9da2-967f-5b15-f0ab-29ed0c7c69a0 (at 10.9.101.70@o2ib4) Oct 05 14:09:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 31c4ea76-8d98-76b3-0d2a-f267092359fe (at 10.9.101.72@o2ib4) Oct 05 14:09:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 98de0dc5-9940-c6b2-2446-ef2f6b2e2bc3 (at 10.9.101.66@o2ib4) Oct 05 14:09:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 14:09:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6b610587-01b4-c16d-f180-8e6dccea1ccc (at 10.9.101.68@o2ib4) Oct 05 14:09:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 14:09:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c7f7b469-0771-3203-e1a4-073b0d6d3f7c (at 10.9.101.63@o2ib4) Oct 05 14:09:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 14:10:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 35a2d3db-6de6-7372-258e-59766d5e315a (at 10.9.101.65@o2ib4) Oct 05 14:10:42 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 14:12:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8e073658-5d71-c10a-9d31-1658501c853d (at 10.9.107.27@o2ib4) Oct 05 14:12:12 oak-md1-s2 kernel: Lustre: Skipped 25 previous similar messages Oct 05 14:14:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f0cb6688-b478-705c-a553-8c0aa06b9237 (at 10.9.106.43@o2ib4) Oct 05 14:14:43 oak-md1-s2 kernel: Lustre: Skipped 31 previous similar messages Oct 05 14:19:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca2911d7-e15b-8887-321a-5c3bb16e7abb (at 10.8.10.22@o2ib6) Oct 05 14:19:51 oak-md1-s2 kernel: Lustre: Skipped 74 previous similar messages Oct 05 14:34:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 85a7c1e3-b517-bf60-fbd3-76cccb0a53bc (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d28647000, cur 1538775299 expire 1538775149 last 1538775072 Oct 05 14:34:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 14:40:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 209b7976-ad55-6bfd-d7b0-952d1ab1c23a (at 10.8.27.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d27800, cur 1538775617 expire 1538775467 last 1538775390 Oct 05 14:53:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2ce6fa59-5e31-71ae-af70-fd0bf18fb38d (at 10.9.104.4@o2ib4) Oct 05 14:53:41 oak-md1-s2 kernel: Lustre: Skipped 35 previous similar messages Oct 05 15:03:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8533e8b-3a11-3aef-6b11-f9c53bf9ac57 (at 10.8.27.21@o2ib6) Oct 05 15:07:44 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 15:07:44 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 05 15:09:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de2e25a6-801f-ad65-eb0b-96c4b5e0a5a6 (at 10.8.31.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edfcc00, cur 1538777341 expire 1538777191 last 1538777114 Oct 05 15:17:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f010ae29-18a8-96eb-6426-4aeac2b4bd6c (at 10.9.115.8@o2ib4) Oct 05 15:19:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 54f505e7-d7a0-c4e3-7e23-13e4d1f96c41 (at 10.8.20.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185d7d7000, cur 1538777948 expire 1538777798 last 1538777721 Oct 05 15:29:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de2e25a6-801f-ad65-eb0b-96c4b5e0a5a6 (at 10.8.31.8@o2ib6) Oct 05 15:29:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 15:32:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fde605ef-5895-8349-000b-5dd26370c22e (at 10.9.105.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039b5a1c00, cur 1538778744 expire 1538778594 last 1538778517 Oct 05 15:37:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbd4bf31-59b4-3233-305a-e8561d53fedf (at 10.8.20.10@o2ib6) Oct 05 15:44:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c579a8b-e621-06a0-fc32-b5110fde9f67 (at 10.8.22.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d22c00, cur 1538779469 expire 1538779319 last 1538779242 Oct 05 15:52:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 29dc3988-bd54-a2fc-7852-0eba7f35ed88 (at 10.8.10.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802a9bbf800, cur 1538779955 expire 1538779805 last 1538779728 Oct 05 15:52:35 oak-md1-s2 kernel: Lustre: Skipped 26 previous similar messages Oct 05 15:54:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 010ca787-4268-12a3-b620-f18a75701351 (at 10.9.105.52@o2ib4) Oct 05 15:58:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f6333ba5-ea61-cf9b-8630-da6242bacbd5 (at 10.8.23.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb82800, cur 1538780337 expire 1538780187 last 1538780110 Oct 05 15:58:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:00:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 83efbddd-c45a-af7b-cfb9-b60b725e3582 (at 10.8.17.21@o2ib6) in 217 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef1400, cur 1538780413 expire 1538780263 last 1538780196 Oct 05 16:00:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:02:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cdea4d19-af50-5605-cf08-fbe5ecdc5fd4 (at 10.8.22.19@o2ib6) Oct 05 16:03:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0c579a8b-e621-06a0-fc32-b5110fde9f67 (at 10.8.22.32@o2ib6) Oct 05 16:03:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 004c122b-02e7-6a9f-39c6-78051b0a1809 (at 10.8.26.27@o2ib6) Oct 05 16:03:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 16:03:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a51c2751-2210-5632-0d46-a3c2ad9d9c6e (at 10.8.22.33@o2ib6) Oct 05 16:03:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a4d6ebc4-1eae-1df9-1e22-87ae53fd9c31 (at 10.8.22.25@o2ib6) Oct 05 16:03:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:04:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8de3f372-d077-3849-ea61-a674d74fdadc (at 10.8.23.36@o2ib6) Oct 05 16:05:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9a4db829-e3cc-45a7-61da-f6e4fb34588d (at 10.9.106.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b76fbb000, cur 1538780700 expire 1538780550 last 1538780473 Oct 05 16:05:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:06:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bd68d4d7-bbad-58da-62a6-c16f42e951b7 (at 10.8.8.13@o2ib6) in 203 seconds. I think it's dead, and I am evicting it. exp ffff881aa9bdc800, cur 1538780776 expire 1538780626 last 1538780573 Oct 05 16:06:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:07:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37867033-7b76-d74e-b7a0-97407136a422 (at 10.8.30.13@o2ib6) in 195 seconds. I think it's dead, and I am evicting it. exp ffff881e25d20000, cur 1538780852 expire 1538780702 last 1538780657 Oct 05 16:08:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2b013eda-077a-097a-0fba-6687e4b3366b (at 10.9.106.18@o2ib4) Oct 05 16:09:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8d15b20b-cce0-3a08-fe85-16ffc43b17df (at 10.8.10.1@o2ib6) Oct 05 16:09:50 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 05 16:16:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10679be3-3dc2-6456-0b34-a512ebb5c982 (at 10.8.10.15@o2ib6) Oct 05 16:16:23 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 05 16:17:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 16649d25-a0da-10c1-381f-0e7ecd4a2744 (at 10.8.4.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186cfe7400, cur 1538781462 expire 1538781312 last 1538781235 Oct 05 16:17:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:22:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30d47dbb-a4ea-0a7d-b10a-4f681473f913 (at 10.9.106.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fecf07800, cur 1538781732 expire 1538781582 last 1538781505 Oct 05 16:22:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:24:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to edf3fbab-bd71-311d-037a-27228294c455 (at 10.8.1.27@o2ib6) Oct 05 16:24:53 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 16:29:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 63fc1a19-8902-02d8-1eac-6758be94d243 (at 10.9.104.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185c29e800, cur 1538782164 expire 1538782014 last 1538781937 Oct 05 16:29:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 16:41:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca7e0dc8-3622-f606-e8c8-5a16f964bc9f (at 10.8.20.25@o2ib6) Oct 05 16:41:35 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 16:42:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 823283f0-28bf-d824-8068-326cad25e72e (at 10.8.20.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c64a000, cur 1538782977 expire 1538782827 last 1538782750 Oct 05 16:42:57 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 16:51:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1534612d-8b6a-e186-b68e-f8a866ebbc9a (at 10.9.104.50@o2ib4) Oct 05 16:51:43 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 17:00:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 926fbb53-fa07-e14a-c303-ad536f948af4 (at 10.9.106.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200be48400, cur 1538784007 expire 1538783857 last 1538783780 Oct 05 17:00:07 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 17:01:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e041b8c9-dace-8861-6540-6a7fcc684186 (at 10.8.20.8@o2ib6) Oct 05 17:01:45 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Oct 05 17:19:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1c92d11a-e2f1-f513-c7f8-2b6130047946 (at 10.8.6.14@o2ib6) Oct 05 17:21:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 192ab630-953f-b6b1-d539-458ab5eae85b (at 10.9.104.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fddf4ec00, cur 1538785265 expire 1538785115 last 1538785038 Oct 05 17:21:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 17:24:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf58d445-a7ca-a616-9976-0f6629ad4709 (at 10.8.2.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805d4fcc000, cur 1538785453 expire 1538785303 last 1538785226 Oct 05 17:24:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 17:37:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9252aca-5530-c0b2-5bb1-087c7dfe179d (at 10.8.26.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0fdc00, cur 1538786228 expire 1538786078 last 1538786001 Oct 05 17:42:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 05 17:42:46 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Oct 05 17:44:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a67cbc4-82d2-f0a8-e6d9-cb0597d49fc7 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88085dc1bc00, cur 1538786687 expire 1538786537 last 1538786460 Oct 05 17:44:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 17:45:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1055a741-3e70-fe8e-c702-1f0da06b99a7 (at 10.9.104.60@o2ib4) Oct 05 17:45:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 17:46:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bbf2b785-df5b-d6f9-a306-2c5d0e2fa7bc (at 10.9.106.25@o2ib4) in 167 seconds. I think it's dead, and I am evicting it. exp ffff881dac936000, cur 1538786763 expire 1538786613 last 1538786596 Oct 05 17:46:03 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 05 17:47:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93cbe953-b4e4-de82-a1cb-b3c30b0a8a1d (at 10.9.106.36@o2ib4) in 224 seconds. I think it's dead, and I am evicting it. exp ffff880f6d1d9c00, cur 1538786839 expire 1538786689 last 1538786615 Oct 05 17:47:19 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 17:48:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c86a420f-c781-b7d5-4291-9e94489b056b (at 10.8.2.16@o2ib6) Oct 05 17:48:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 17:49:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a97940aa-51a8-4978-6215-7edb585ce8d4 (at 10.8.25.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b29bfec00, cur 1538786994 expire 1538786844 last 1538786767 Oct 05 17:49:54 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 17:56:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f9252aca-5530-c0b2-5bb1-087c7dfe179d (at 10.8.26.6@o2ib6) Oct 05 18:01:24 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 18:06:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8abfc2ec-549c-7f76-0e8b-0bde95603894 (at 10.8.23.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28c400, cur 1538787988 expire 1538787838 last 1538787761 Oct 05 18:06:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 18:06:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 90601611-f7cd-ad01-4385-1cd22e98279c (at 10.8.8.31@o2ib6) Oct 05 18:06:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 18:18:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ed70a58-1168-d533-9c28-a61cd79d98ac (at 10.8.20.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d26c00, cur 1538788691 expire 1538788541 last 1538788464 Oct 05 18:18:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 18:22:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 05 18:22:13 oak-md1-s2 kernel: Lustre: Skipped 17 previous similar messages Oct 05 18:36:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a11f4956-fece-bb34-a9e4-e60af816ee2c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d7cb400, cur 1538789788 expire 1538789638 last 1538789561 Oct 05 18:36:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43038ec1-e22e-886f-2a71-7f475d6cef88 (at 10.8.20.24@o2ib6) Oct 05 18:36:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 18:54:02 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 05 18:54:02 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 05 19:02:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 07824a49-3a84-1f19-2ca8-074eac510d03 (at 10.8.4.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809762eac00, cur 1538791363 expire 1538791213 last 1538791136 Oct 05 19:25:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf95c8d8-9bec-0afd-6531-8cf1b4687200 (at 10.9.106.35@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039235800, cur 1538792751 expire 1538792601 last 1538792524 Oct 05 19:49:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 03f570ee-a890-7edd-e1d9-c0b01134b662 (at 10.9.106.35@o2ib4) Oct 05 19:52:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c27341d6-aaac-e004-3a5c-a89425f570a1 (at 10.8.26.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880041c1a400, cur 1538794357 expire 1538794207 last 1538794130 Oct 05 19:53:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 76301ed5-eb29-4f15-835e-4e69c8ceb2f4 (at 10.8.10.26@o2ib6) in 223 seconds. I think it's dead, and I am evicting it. exp ffff88000c64b400, cur 1538794433 expire 1538794283 last 1538794210 Oct 05 19:58:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e8b809b5-cab7-bc4c-0fce-84961e130a45 (at 10.9.102.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fec3d0400, cur 1538794714 expire 1538794564 last 1538794487 Oct 05 19:58:34 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Oct 05 20:08:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 54e881d2-8667-ffe3-49e9-519efc6e4941 (at 10.8.10.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee604ac00, cur 1538795339 expire 1538795189 last 1538795112 Oct 05 20:10:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e1a07fbe-fc23-5c7d-feab-046bf9b7117d (at 10.8.22.16@o2ib6) in 221 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e47c00, cur 1538795415 expire 1538795265 last 1538795194 Oct 05 20:10:15 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 20:12:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7b57bffd-9dc4-b1cb-17fd-e78b7b2589f8 (at 10.8.31.10@o2ib6) Oct 05 20:12:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c27341d6-aaac-e004-3a5c-a89425f570a1 (at 10.8.26.3@o2ib6) Oct 05 20:13:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7f488842-06e4-96d4-ad37-762861415824 (at 10.8.21.10@o2ib6) Oct 05 20:13:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 20:14:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3dfc8037-f91d-2f30-5f64-c0f9e53622c6 (at 10.8.30.29@o2ib6) Oct 05 20:16:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d3c4380-9608-a655-5eb7-af49e4a2381a (at 10.8.4.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c589c00, cur 1538795774 expire 1538795624 last 1538795547 Oct 05 20:17:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 05 20:21:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e8b809b5-cab7-bc4c-0fce-84961e130a45 (at 10.9.102.55@o2ib4) Oct 05 20:21:16 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 20:21:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb8a76b3-e23b-f206-2acf-167680c9d65a (at 10.9.104.71@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db1f20800, cur 1538796092 expire 1538795942 last 1538795865 Oct 05 20:21:32 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 05 20:28:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e1a07fbe-fc23-5c7d-feab-046bf9b7117d (at 10.8.22.16@o2ib6) Oct 05 20:31:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 15e3e32e-edb9-4dfb-2462-cc9eaac285ce (at 10.9.101.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818893ed800, cur 1538796671 expire 1538796521 last 1538796444 Oct 05 20:31:11 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 20:40:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6e7dbe8-678f-5bec-58d6-4a582a6d69c2 (at 10.8.4.29@o2ib6) Oct 05 20:40:16 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 20:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 64311bbc-e9dc-a22f-aed1-4d2b419adaa9 (at 10.8.25.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef1c00, cur 1538797389 expire 1538797239 last 1538797162 Oct 05 20:52:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0875cde5-2eb7-0c4c-d39f-7591125261c9 (at 10.9.101.18@o2ib4) Oct 05 20:52:48 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 05 21:02:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 18ae548d-a721-f02f-c9b8-aa5a7c37e274 (at 10.8.12.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edf9400, cur 1538798547 expire 1538798397 last 1538798320 Oct 05 21:03:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 58bcf75f-4fde-4a3f-5845-4b3c5e7fe6e9 (at 10.9.104.30@o2ib4) Oct 05 21:03:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 21:05:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c1ec48d-c522-a534-0a29-da9ed4b87f40 (at 10.9.104.30@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881906265800, cur 1538798702 expire 1538798552 last 1538798475 Oct 05 21:05:02 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 05 21:09:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 72ac381f-3e5a-1809-d70e-0dcaed1c54da (at 10.8.31.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c870800, cur 1538798950 expire 1538798800 last 1538798723 Oct 05 21:12:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28f5182f-d382-7db6-e2d8-b5745a37e02f (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88021fc20000, cur 1538799158 expire 1538799008 last 1538798931 Oct 05 21:12:38 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 21:16:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 786fe993-d3bf-e5cd-64b6-ebb6fa2963cd (at 10.8.20.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e7000, cur 1538799374 expire 1538799224 last 1538799147 Oct 05 21:16:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 21:20:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b2f2d2d0-c49a-d136-5b25-3d04b42cc961 (at 10.8.23.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b723c00, cur 1538799638 expire 1538799488 last 1538799411 Oct 05 21:20:38 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 05 21:24:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a8ff1add-9a8e-bd28-9e25-1a5bbe2d7e39 (at 10.8.30.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd65b800, cur 1538799846 expire 1538799696 last 1538799619 Oct 05 21:24:06 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 05 21:26:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2104b814-a56f-f122-8c9f-eb491e58982b (at 10.8.12.33@o2ib6) Oct 05 21:27:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f6eef988-a7c8-6d09-0455-c99dabce238f (at 10.8.12.20@o2ib6) Oct 05 21:27:35 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 05 21:31:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e2c4effc-a6db-b02c-b57d-9e817194b85a (at 10.8.25.9@o2ib6) Oct 05 21:31:38 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 05 21:32:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74ab736e-280d-1784-9a2e-115c5e1e592c (at 10.8.25.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1b400, cur 1538800376 expire 1538800226 last 1538800149 Oct 05 21:34:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3ca24efb-3e7a-c90f-13db-68413780c893 (at 10.8.16.8@o2ib6) in 204 seconds. I think it's dead, and I am evicting it. exp ffff880eda289c00, cur 1538800452 expire 1538800302 last 1538800248 Oct 05 21:37:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 199c6734-f713-bc59-1bdf-f99359134a9a (at 10.8.20.34@o2ib6) Oct 05 21:37:30 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 05 21:51:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 74ab736e-280d-1784-9a2e-115c5e1e592c (at 10.8.25.32@o2ib6) Oct 05 21:51:27 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 05 22:16:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 221e4d45-c333-dc61-bb54-9c6a6ad3c89f (at 10.9.114.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272375400, cur 1538803016 expire 1538802866 last 1538802789 Oct 05 22:20:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b408e970-265e-551d-bf93-f064ae5ef22e (at 10.8.24.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee604c000, cur 1538803204 expire 1538803054 last 1538802977 Oct 05 22:20:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 22:32:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94ca5020-561c-0e63-7743-b03f5cc407ae (at 10.9.103.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef28dcc00, cur 1538803940 expire 1538803790 last 1538803713 Oct 05 22:32:20 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 22:38:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ebb1f52-6094-2c65-f0d1-841ac9304cdb (at 10.8.24.12@o2ib6) Oct 05 22:38:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 22:40:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dcf92cac-ee35-5d93-76bf-854632ffdf48 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880995a96000, cur 1538804414 expire 1538804264 last 1538804187 Oct 05 22:56:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 94ca5020-561c-0e63-7743-b03f5cc407ae (at 10.9.103.7@o2ib4) Oct 05 22:56:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 05 23:25:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7932e2d6-5f11-b3eb-320c-e26a25b98d9f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004a19dc00, cur 1538807120 expire 1538806970 last 1538806893 Oct 05 23:30:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5aaa631e-fdbf-074e-ce17-ec0a6a693d22 (at 10.9.101.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880180b59400, cur 1538807415 expire 1538807265 last 1538807188 Oct 05 23:34:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9baf9f95-2a18-bcd2-1272-f3710ca63d76 (at 10.9.101.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818973f8400, cur 1538807669 expire 1538807519 last 1538807442 Oct 05 23:34:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 05 23:34:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 05 23:40:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d6080864-a3b6-41a8-2c71-7cc47b358c4b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88952000, cur 1538808037 expire 1538807887 last 1538807810 Oct 05 23:41:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 365b4dbc-9ba6-de78-b4b4-e8b7c3309b0c (at 10.8.1.29@o2ib6) in 209 seconds. I think it's dead, and I am evicting it. exp ffff881dacc5ac00, cur 1538808113 expire 1538807963 last 1538807904 Oct 05 23:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 686f694a-82ac-93e2-6170-d8fc9237f2c4 (at 10.8.21.6@o2ib6) in 215 seconds. I think it's dead, and I am evicting it. exp ffff881890e48c00, cur 1538808189 expire 1538808039 last 1538807974 Oct 05 23:44:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 05 23:49:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 05 23:50:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2219ba16-2ab9-6279-2dfa-83e04b252e69 (at 10.8.22.2@o2ib6) Oct 05 23:50:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 133b0425-601f-d62d-a28b-1fdab1443b30 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e95c9cc00, cur 1538808619 expire 1538808469 last 1538808392 Oct 05 23:50:19 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 05 23:51:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5aaa631e-fdbf-074e-ce17-ec0a6a693d22 (at 10.9.101.46@o2ib4) Oct 05 23:53:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 25b0c9d6-3ff9-e87b-7002-bfe82c429fb0 (at 10.8.26.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808d6af3000, cur 1538808790 expire 1538808640 last 1538808563 Oct 05 23:57:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 05 23:57:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b19fc033-da35-f9fc-d637-b4210704fb78 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f96a50000, cur 1538809056 expire 1538808906 last 1538808829 Oct 05 23:57:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 00:00:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 58217d2b-ce54-41ec-d520-c8acb4a59b6d (at 10.8.30.3@o2ib6) Oct 06 00:00:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 00:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 813689bc-24ca-1be0-922d-7b5ef11dc6b9 (at 10.8.24.29@o2ib6) Oct 06 00:02:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2bd19781-7eb2-1df9-c901-19d0846ba5e9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6d1dd000, cur 1538809360 expire 1538809210 last 1538809133 Oct 06 00:07:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 00:07:15 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 06 00:12:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 00:12:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 00:13:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d8d55486-7ff1-4ec0-bcd8-300542ccdc9a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c5a3400, cur 1538810037 expire 1538809887 last 1538809810 Oct 06 00:13:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 00:24:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 00:24:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 00:25:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39fd4c88-5491-b7e4-ece0-aefdcc3d8c12 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881846f58000, cur 1538810720 expire 1538810570 last 1538810493 Oct 06 00:25:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 00:36:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42995b25-a93e-f97e-f7c7-047e7f6d990d (at 10.8.25.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8c9c00, cur 1538811387 expire 1538811237 last 1538811160 Oct 06 00:36:27 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Oct 06 00:45:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80099489-3226-75b7-a938-9c2e172b6eec (at 10.8.26.15@o2ib6) Oct 06 00:45:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 00:48:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 33ebe196-ac4c-d20e-bc50-742494e7f18e (at 10.8.22.10@o2ib6) Oct 06 00:51:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a82fe5d-1e14-35b7-1533-3c3ff2c40621 (at 10.9.101.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200b3d8800, cur 1538812264 expire 1538812114 last 1538812037 Oct 06 00:54:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 87555990-c380-af2a-c021-a753301f8ad2 (at 10.8.25.8@o2ib6) Oct 06 00:54:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 01:06:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9fcec27-7776-0ec0-9741-b74e916bcb54 (at 10.9.104.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184d7adc00, cur 1538813163 expire 1538813013 last 1538812936 Oct 06 01:12:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a82fe5d-1e14-35b7-1533-3c3ff2c40621 (at 10.9.101.6@o2ib4) Oct 06 01:12:35 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 06 01:28:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c94d08be-0c05-348d-22c7-f38336d0d93a (at 10.9.104.52@o2ib4) Oct 06 01:30:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a61dbfaf-5672-d65b-9f59-e818b81bb176 (at 10.9.101.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816e53b9000, cur 1538814616 expire 1538814466 last 1538814389 Oct 06 01:50:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 590c101f-9072-5fe4-67f6-9ef5670ade95 (at 10.8.24.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d24000, cur 1538815824 expire 1538815674 last 1538815597 Oct 06 01:52:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9b520a7-2fe2-16e1-ae5c-5ab68053285a (at 10.9.101.2@o2ib4) Oct 06 02:05:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 007389d1-dd5d-3121-1639-b39dd0cfdaf0 (at 10.8.30.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0fc000, cur 1538816752 expire 1538816602 last 1538816525 Oct 06 02:09:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f2b8f9b-76f7-7227-b04c-66cd3afe1f32 (at 10.9.101.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185a3fa800, cur 1538816941 expire 1538816791 last 1538816714 Oct 06 02:09:01 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 06 02:09:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 86e301e8-a840-cff3-1355-933f8c7263d9 (at 10.8.24.26@o2ib6) Oct 06 02:18:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2db09ff5-78bf-7486-5d09-181ca16dfb20 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210b000, cur 1538817503 expire 1538817353 last 1538817276 Oct 06 02:20:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1d466413-418e-db82-5337-6a2f078692ec (at 10.8.19.2@o2ib6) Oct 06 02:20:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ddff929a-cc60-fb27-9b8b-1a40efc890e0 (at 10.8.19.3@o2ib6) Oct 06 02:23:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2c8ed3ab-61a9-5456-3505-ea9bd42f7f2a (at 10.8.22.8@o2ib6) Oct 06 02:24:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bc53ecd9-06fb-fb05-a055-b87173ddf19a (at 10.8.22.7@o2ib6) Oct 06 02:24:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e5ea46dc-4dfd-9ce5-49ca-f77bd0aaba15 (at 10.8.26.28@o2ib6) Oct 06 02:24:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9e870f8-bb41-d02c-ebc5-fe24e2ab6a99 (at 10.8.13.29@o2ib6) Oct 06 02:24:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 02:25:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 007389d1-dd5d-3121-1639-b39dd0cfdaf0 (at 10.8.30.28@o2ib6) Oct 06 02:30:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7896719e-213b-225c-de9a-0b3a69876314 (at 10.9.101.53@o2ib4) Oct 06 02:38:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7d4433d8-921c-0a7c-3ccb-26ba1ef18d63 (at 10.8.9.4@o2ib6) Oct 06 02:52:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65a615ce-7349-f25d-5432-727b42fb2b10 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200befc800, cur 1538819558 expire 1538819408 last 1538819331 Oct 06 03:23:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 502470e0-0491-2384-d5c7-38f508ec5468 (at 10.9.106.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da1087400, cur 1538821413 expire 1538821263 last 1538821186 Oct 06 03:42:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e1041d5a-cba1-45a2-1676-4eeb820ad4c1 (at 10.8.21.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810302f2800, cur 1538822554 expire 1538822404 last 1538822327 Oct 06 03:47:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 26ce7643-66b7-173d-e38b-3a41e32c0bad (at 10.9.106.20@o2ib4) Oct 06 03:49:17 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 03:49:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7388341c-2608-22d6-71f7-6770666c2f01 (at 10.8.23.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea512b400, cur 1538822995 expire 1538822845 last 1538822768 Oct 06 03:49:55 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 06 03:56:49 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 03:56:49 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 06 03:57:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd6dfe72-84f6-8f65-aa91-baf4c46d180c (at 10.8.26.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880041c1e800, cur 1538823427 expire 1538823277 last 1538823200 Oct 06 03:57:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 04:01:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7e08b657-b37b-5a11-d095-6dd564fb1792 (at 10.8.23.3@o2ib6) Oct 06 04:01:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6afb722f-18f7-2e65-8948-844c06f3b28e (at 10.8.22.36@o2ib6) Oct 06 04:01:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 04:01:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4429ee1c-6fc6-da0d-0a60-3406a6bb91ac (at 10.8.23.4@o2ib6) Oct 06 04:01:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aa36fc16-505e-a9e9-38b1-6d2cc8b192a1 (at 10.8.21.19@o2ib6) Oct 06 04:01:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 47fbb9df-d3eb-ba37-c069-782ba5c2e44e (at 10.8.21.15@o2ib6) Oct 06 04:01:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 04:03:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 71e62346-58b7-df7a-d804-a5ef863aee65 (at 10.8.20.35@o2ib6) Oct 06 04:08:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 203b773a-81c3-b207-3bf4-bf1c8a54f05d (at 10.8.23.25@o2ib6) Oct 06 04:16:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9580d2f-2068-c20e-f3b3-2172b32533fa (at 10.8.26.19@o2ib6) Oct 06 04:16:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 04:29:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3f49fbe-cf93-31e2-95b7-f629b719d389 (at 10.9.103.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b63d24800, cur 1538825344 expire 1538825194 last 1538825117 Oct 06 04:29:04 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 06 04:32:08 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 04:32:08 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 06 04:35:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb8f659c-2fcc-8841-36b7-04aabeda7e49 (at 10.8.25.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afc400, cur 1538825718 expire 1538825568 last 1538825491 Oct 06 04:35:34 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 04:41:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 04:41:50 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 06 04:42:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ff82877-6a81-2e96-dcec-1504dfdd902c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805f1816800, cur 1538826148 expire 1538825998 last 1538825921 Oct 06 04:42:28 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 04:49:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39f6cd08-0284-b556-faa9-d1cfe9f44894 (at 10.9.101.36@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b275a8000, cur 1538826593 expire 1538826443 last 1538826366 Oct 06 04:51:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7de016a8-cc59-8081-bff4-fcb7df8ad2f4 (at 10.8.18.35@o2ib6) in 159 seconds. I think it's dead, and I am evicting it. exp ffff880fe9df3800, cur 1538826669 expire 1538826519 last 1538826510 Oct 06 04:51:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d3f49fbe-cf93-31e2-95b7-f629b719d389 (at 10.9.103.13@o2ib4) Oct 06 04:53:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb8f659c-2fcc-8841-36b7-04aabeda7e49 (at 10.8.25.4@o2ib6) Oct 06 04:53:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 04:53:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3f950878-40ba-d56f-6cf2-f68d02a8791d (at 10.9.101.30@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d894a0400, cur 1538826838 expire 1538826688 last 1538826611 Oct 06 05:03:44 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 05:03:44 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 06 05:13:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ba6899e7-9708-ca4b-aa2b-90ce54246562 (at 10.9.101.36@o2ib4) Oct 06 05:13:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 05:13:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0137a2d-0566-84ac-9549-252389debee9 (at 10.9.101.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851230800, cur 1538828032 expire 1538827882 last 1538827805 Oct 06 05:15:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 291d7bea-f250-4d15-130f-a2b94cc43882 (at 10.9.101.30@o2ib4) Oct 06 05:34:21 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 05:35:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 69bc8ae0-d560-1e7e-c335-63319ac6c4e2 (at 10.9.101.33@o2ib4) Oct 06 06:16:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f69a17f-20e8-32e4-a8ab-7ae752f9de10 (at 10.8.22.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185d7d0c00, cur 1538831804 expire 1538831654 last 1538831577 Oct 06 06:21:11 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 06:25:02 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 06:33:38 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 06:35:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f69a17f-20e8-32e4-a8ab-7ae752f9de10 (at 10.8.22.35@o2ib6) Oct 06 06:35:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 363fdcba-500f-edf1-52a8-fb7a1245467b (at 10.8.22.29@o2ib6) Oct 06 06:50:44 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 06:50:45 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 07:03:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5853f097-8cc2-6e11-a360-1c4075e09668 (at 10.8.25.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005dcb0000, cur 1538834619 expire 1538834469 last 1538834392 Oct 06 07:03:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 07:22:38 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 07:22:38 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 06 07:22:39 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 07:32:24 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 07:32:55 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 07:50:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8cda5002-692f-7ae5-5d9f-a5cc4b186aba (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa904400, cur 1538837454 expire 1538837304 last 1538837227 Oct 06 07:54:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30deb903-da29-b115-d7b1-9ef0edf960c2 (at 10.8.20.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4b000, cur 1538837659 expire 1538837509 last 1538837432 Oct 06 07:59:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 26d02c33-7123-3f9f-34a5-25afade49aa4 (at 10.9.101.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa970400, cur 1538837966 expire 1538837816 last 1538837739 Oct 06 08:00:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c5c13bb-947d-b28e-eb21-aaec3da3c864 (at 10.8.10.11@o2ib6) in 178 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8c8000, cur 1538838042 expire 1538837892 last 1538837864 Oct 06 08:12:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7a29b7ea-27b6-3537-6761-540c92a0a76e (at 10.8.20.31@o2ib6) Oct 06 08:21:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b54aa4b4-f2cc-6feb-c3eb-1faa5de88028 (at 10.9.101.16@o2ib4) Oct 06 08:27:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eb35b607-6e5e-9982-db36-09536c6deedd (at 10.8.10.11@o2ib6) Oct 06 08:34:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a7b3f866-0c39-0fe1-4cb7-cd02f2ec2184 (at 10.12.4.35@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4b597400, cur 1538840056 expire 1538839906 last 1538839829 Oct 06 08:35:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa5541c6-b49f-97f9-7ebd-59eaa5151787 (at 10.12.4.35@o2ib) Oct 06 08:45:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e211274-1447-a4b8-012f-c30033ac82a0 (at 10.8.25.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee604c800, cur 1538840752 expire 1538840602 last 1538840525 Oct 06 08:45:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 08:52:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 03389481-061b-fff3-409d-ef891f38f459 (at 10.12.4.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc6d000, cur 1538841169 expire 1538841019 last 1538840942 Oct 06 08:53:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b2e21f8-1922-74ab-dec7-a4e0da7ed051 (at 10.12.4.49@o2ib) Oct 06 08:54:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04bc9d04-ffe3-bc35-f637-b3752a5d8f66 (at 10.8.31.9@o2ib6) in 220 seconds. I think it's dead, and I am evicting it. exp ffff88170039dc00, cur 1538841245 expire 1538841095 last 1538841025 Oct 06 08:56:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 54e43e8a-8a2a-c295-71f7-f6fa71938cba (at 10.9.101.27@o2ib4) Oct 06 09:04:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 33ec7d14-30b3-1a37-08d4-0b6b1c687758 (at 10.8.25.25@o2ib6) Oct 06 09:12:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 04bc9d04-ffe3-bc35-f637-b3752a5d8f66 (at 10.8.31.9@o2ib6) Oct 06 09:15:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f0fa3529-f755-3e72-9b64-1b87cc07d33d (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac8fe800, cur 1538842538 expire 1538842388 last 1538842311 Oct 06 09:31:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c40808fa-a89a-ff5b-2c9d-153f7d1389a6 (at 10.9.113.6@o2ib4) Oct 06 09:33:20 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 09:33:20 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 06 10:06:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c6fca84b-6f6a-0d78-aa5e-5a836dfdac3a (at 10.12.4.83@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015ca4800, cur 1538845597 expire 1538845447 last 1538845370 Oct 06 10:07:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to acdd75fe-b325-54ad-3e7e-2cce9f7641e3 (at 10.12.4.83@o2ib) Oct 06 10:15:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 003dbe9c-b98b-b13c-9d37-1c4c71cb69ea (at 10.9.103.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a275400, cur 1538846116 expire 1538845966 last 1538845889 Oct 06 10:23:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5532e964-0444-43cd-bb86-8baf04ed50d4 (at 10.8.9.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d2690400, cur 1538846583 expire 1538846433 last 1538846356 Oct 06 10:28:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb3f439d-94e4-73c3-abf8-654c72833745 (at 10.9.101.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058c460c00, cur 1538846887 expire 1538846737 last 1538846660 Oct 06 10:37:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 003dbe9c-b98b-b13c-9d37-1c4c71cb69ea (at 10.9.103.4@o2ib4) Oct 06 10:44:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09224f28-d4e9-41eb-4826-893251222ec0 (at 10.8.9.10@o2ib6) Oct 06 10:51:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 713d61b5-31b3-574c-a394-58d6045b6f73 (at 10.9.101.8@o2ib4) Oct 06 10:57:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 57159387-3baf-046b-f0af-f78da67eb6e3 (at 10.8.26.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880109889800, cur 1538848641 expire 1538848491 last 1538848414 Oct 06 11:18:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 57159387-3baf-046b-f0af-f78da67eb6e3 (at 10.8.26.13@o2ib6) Oct 06 11:42:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4890b1a-23c3-f8e4-fe1f-ab8dc8040dbc (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e151ea800, cur 1538851376 expire 1538851226 last 1538851149 Oct 06 11:48:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71818191-9fe0-b007-c88c-50feb4103277 (at 10.8.25.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af6800, cur 1538851692 expire 1538851542 last 1538851465 Oct 06 11:51:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b2153cc-9f8b-630d-a564-24d17f4bbdf1 (at 10.9.106.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bef730c00, cur 1538851868 expire 1538851718 last 1538851641 Oct 06 11:51:08 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 06 12:01:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a539c176-9c88-cd1d-5bb4-8b5fe5f5d3db (at 10.8.12.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4c800, cur 1538852489 expire 1538852339 last 1538852262 Oct 06 12:01:29 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 12:03:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 06 12:06:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3c2be6d-fc11-a24a-2b76-1e35df502816 (at 10.8.25.20@o2ib6) Oct 06 12:06:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 58f37b89-7b9c-76d3-6ceb-323d8a1ba23a (at 10.8.25.18@o2ib6) Oct 06 12:06:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cd0a10ce-e6e4-dcd4-d9ca-cab57b232b06 (at 10.8.30.34@o2ib6) Oct 06 12:06:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 12:07:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 40a704e7-091c-48a5-4270-6046c349ab66 (at 10.8.25.17@o2ib6) Oct 06 12:07:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 12:09:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d8c9b12-b934-725d-aee3-9111c1723937 (at 10.8.30.33@o2ib6) Oct 06 12:12:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 796bca70-d9ba-bf1d-1948-d99be3270cf3 (at 10.8.11.17@o2ib6) Oct 06 12:12:54 oak-md1-s2 kernel: LustreError: 4653:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.40@o2ib4) failed to reply to blocking AST (req@ffff88179d78ec00 x1610630863250816 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff88179ab72200/0x6c00c3fee93445db lrc: 5/0,0 mode: PR/PR res: [0x200017cf3:0x16d89:0x0].0x0 bits 0x1b rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.9.105.40@o2ib4 remote: 0xba230d6e90aa940 expref: 52873 pid: 10557 timeout: 0 lvb_type: 0 Oct 06 12:12:54 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.40@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Oct 06 12:14:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 838a1146-4ce3-4aa9-a3f0-195a20e5b866 (at 10.9.106.56@o2ib4) Oct 06 12:24:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2101118f-22b1-a29c-ce80-4cf34cb17960 (at 10.9.101.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2112400, cur 1538853841 expire 1538853691 last 1538853614 Oct 06 12:24:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 12:24:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 24c3fca1-67d3-2aba-607e-e27143a5de35 (at 10.8.12.14@o2ib6) Oct 06 12:24:53 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 12:27:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 332f6649-3dbb-5767-0579-f02c8eca5dca (at 10.9.101.49@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb83400, cur 1538854042 expire 1538853892 last 1538853815 Oct 06 12:27:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 12:27:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b8ee8308-af7d-93c0-31de-7113419a43a7 (at 10.8.11.24@o2ib6) Oct 06 12:33:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7084cd75-5961-f8dd-1f74-e3b43a949990 (at 10.9.101.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe69dcc00, cur 1538854430 expire 1538854280 last 1538854203 Oct 06 12:33:50 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 06 12:45:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4a9b874e-3152-6d10-33c8-b00f68018d2f (at 10.9.101.7@o2ib4) Oct 06 12:47:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 745fd9a9-4789-0cf2-6430-26c90ed54e2e (at 10.9.101.3@o2ib4) Oct 06 12:47:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 12:49:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4340ff4d-7982-f3ce-a4c5-83270076899c (at 10.9.101.39@o2ib4) Oct 06 12:51:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d5e55310-4d61-f0b1-f2fe-c961b8a421b3 (at 10.9.101.40@o2ib4) Oct 06 12:51:36 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 12:52:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a43e3262-b8f2-393b-3043-684be986c232 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b7ae4a400, cur 1538855520 expire 1538855370 last 1538855293 Oct 06 12:56:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 21d6fdc2-238b-fea6-0e18-6a165c1ea373 (at 10.9.101.13@o2ib4) Oct 06 12:56:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 12:57:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 91efb86d-339e-4c4d-6f3f-7eb68f22c3a0 (at 10.8.4.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9443c00, cur 1538855830 expire 1538855680 last 1538855603 Oct 06 12:57:10 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 13:11:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e713af16-1822-2af3-ee58-66bace32f16b (at 10.8.22.21@o2ib6) Oct 06 13:21:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2eb5c032-e495-d959-c1b3-624fdefab331 (at 10.8.4.33@o2ib6) Oct 06 13:21:13 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 06 13:55:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 13:55:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 13:56:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d809bdc1-c5d2-7edc-0fa8-9ec0e6712cca (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881520e6e800, cur 1538859388 expire 1538859238 last 1538859161 Oct 06 13:56:28 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 06 14:04:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 14:04:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b302e91-2728-570d-f384-96eb0de81584 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1a400, cur 1538859884 expire 1538859734 last 1538859657 Oct 06 14:12:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 14:13:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c029eaba-97a8-e136-012d-76ce7906c0fc (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe97c2400, cur 1538860400 expire 1538860250 last 1538860173 Oct 06 14:37:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a5bb1c6b-6c6b-d109-de44-7f2fb24e9c22 (at 10.8.13.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda288000, cur 1538861828 expire 1538861678 last 1538861601 Oct 06 15:00:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 23516c40-6752-1a5d-abee-85000b467993 (at 10.8.13.2@o2ib6) Oct 06 15:18:17 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 15:18:18 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 15:23:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 148b8a76-5915-f5d5-97aa-f801fc043a9c (at 10.8.24.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c877400, cur 1538864623 expire 1538864473 last 1538864396 Oct 06 15:42:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 638aad98-5b33-adf2-79e9-cf35e2fa1ad5 (at 10.8.24.16@o2ib6) Oct 06 15:42:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 15ff845f-f011-1072-bd8c-46ffec445fe4 (at 10.8.24.23@o2ib6) Oct 06 15:42:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 16:12:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05ff173f-6631-9b3a-91e5-7598e300de0b (at 10.8.25.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c871000, cur 1538867554 expire 1538867404 last 1538867327 Oct 06 16:12:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 16:31:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 05ff173f-6631-9b3a-91e5-7598e300de0b (at 10.8.25.21@o2ib6) Oct 06 16:35:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f326293a-8baf-cf3b-cedd-b8167fcdc711 (at 10.8.17.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef3000, cur 1538868934 expire 1538868784 last 1538868707 Oct 06 16:35:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 16:36:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55ebe593-d379-af9d-f2d9-3fb932e423ba (at 10.8.20.11@o2ib6) in 193 seconds. I think it's dead, and I am evicting it. exp ffff88082edfd400, cur 1538869010 expire 1538868860 last 1538868817 Oct 06 16:37:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a658811-3d30-68a8-9254-43d6be4303df (at 10.8.11.23@o2ib6) Oct 06 16:53:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dcf75eba-1006-ff57-1378-02d2632c499a (at 10.8.17.9@o2ib6) Oct 06 16:55:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 55ebe593-d379-af9d-f2d9-3fb932e423ba (at 10.8.20.11@o2ib6) Oct 06 17:03:38 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 06 17:09:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bfc9d24-a347-d419-b237-7e4889c0952f (at 10.8.23.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee604d800, cur 1538870962 expire 1538870812 last 1538870735 Oct 06 17:13:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa254f4c-2b7e-cab3-ec49-25470e98baf1 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b76fbd000, cur 1538871212 expire 1538871062 last 1538870985 Oct 06 17:13:32 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 06 17:27:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7ee64c2f-c137-ef43-af31-6ea0dd54fcce (at 10.8.20.7@o2ib6) Oct 06 17:27:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30158d38-6dab-d2c0-e4cc-d9cfb76a7a73 (at 10.8.30.5@o2ib6) Oct 06 17:28:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 17b6f26e-76e4-3e11-e675-5c4acf73690f (at 10.8.30.21@o2ib6) Oct 06 17:30:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3bfc9d24-a347-d419-b237-7e4889c0952f (at 10.8.23.32@o2ib6) Oct 06 18:01:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ed91b3ac-5c77-ff2a-ec59-5f4c5dd8a580 (at 10.8.12.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e5000, cur 1538874091 expire 1538873941 last 1538873864 Oct 06 18:02:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 16b90319-ed52-b805-3c60-991bf4b10fa5 (at 10.8.1.30@o2ib6) in 152 seconds. I think it's dead, and I am evicting it. exp ffff880e3875f000, cur 1538874167 expire 1538874017 last 1538874015 Oct 06 18:05:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a1b7f1b8-e64d-60f2-4596-0d8716d837e7 (at 10.9.106.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab2f40400, cur 1538874345 expire 1538874195 last 1538874118 Oct 06 18:09:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 428e49bd-b81e-538c-9e12-fb88486e14bf (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1f800, cur 1538874564 expire 1538874414 last 1538874337 Oct 06 18:13:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0381ade2-0c5c-6659-1b6d-65a3e607a757 (at 10.8.2.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f927e6c00, cur 1538874839 expire 1538874689 last 1538874612 Oct 06 18:13:59 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 06 18:28:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a17ecf4-f05f-9d51-1ac8-09426d510b17 (at 10.8.1.30@o2ib6) Oct 06 18:33:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ac7c6cfa-9383-b165-1cd0-a6c46d94986d (at 10.9.106.3@o2ib4) Oct 06 18:33:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d759553e-e7f9-8ea2-590f-0489493b9d2c (at 10.8.11.11@o2ib6) Oct 06 18:33:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 56d454d8-9b1f-e57c-46a4-07845d876de0 (at 10.8.11.8@o2ib6) Oct 06 18:34:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea70c050-23ba-10a9-b9c6-ac0c7350e680 (at 10.8.11.5@o2ib6) Oct 06 18:34:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 771a2d77-7dd6-9225-b9ca-aa78fb95360a (at 10.8.11.19@o2ib6) Oct 06 18:34:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 18:35:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 428e49bd-b81e-538c-9e12-fb88486e14bf (at 10.8.11.9@o2ib6) Oct 06 18:35:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 18:38:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f41fcf7c-44d1-6c82-6057-02cf3b2d4405 (at 10.8.2.21@o2ib6) Oct 06 18:45:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 29acfff8-c6bc-c7f2-c78e-cbc89b788df7 (at 10.8.12.23@o2ib6) Oct 06 18:58:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df57da30-ec83-2454-9a0b-14b85a74f9d4 (at 10.9.105.40@o2ib4) Oct 06 18:59:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8b9a0775-6c8a-d66f-edcb-bf52652f5756 (at 10.9.105.40@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93137000, cur 1538877566 expire 1538877416 last 1538877339 Oct 06 19:00:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d53fb913-e5eb-657d-defe-972bd51961b6 (at 10.9.106.1@o2ib4) in 213 seconds. I think it's dead, and I am evicting it. exp ffff880f59a41400, cur 1538877642 expire 1538877492 last 1538877429 Oct 06 19:28:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 506036e0-2e3e-5387-9fd0-edec1850bdc4 (at 10.9.106.1@o2ib4) Oct 06 19:41:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 19:41:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 009cda78-1ab0-b3d5-afcc-107743c616ed (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db0387400, cur 1538880118 expire 1538879968 last 1538879891 Oct 06 19:46:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 19:47:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75f44d83-9f15-37b0-c2a6-e708877c8adb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184eb1c000, cur 1538880432 expire 1538880282 last 1538880205 Oct 06 19:52:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 19:53:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c826e9b-d9a2-10fd-3db0-5317a13866cf (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88085dc19800, cur 1538880813 expire 1538880663 last 1538880586 Oct 06 19:57:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 19:58:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c8920b0-7bfc-50f2-8398-8feae49b9e14 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811eab23c00, cur 1538881105 expire 1538880955 last 1538880878 Oct 06 20:03:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 20:04:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b3c5628-93e9-ea2b-5a29-d02d9388e722 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a16a86400, cur 1538881446 expire 1538881296 last 1538881219 Oct 06 20:08:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 20:09:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f728d0b2-69d9-5b8a-ce74-28c679a0503c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272372800, cur 1538881757 expire 1538881607 last 1538881530 Oct 06 20:14:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 20:14:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38e6f790-da24-b8ae-6599-3ad8364b9e65 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186c670800, cur 1538882085 expire 1538881935 last 1538881858 Oct 06 20:16:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b18402f1-264f-3ec1-27fd-4280026b82c7 (at 10.8.11.36@o2ib6) in 202 seconds. I think it's dead, and I am evicting it. exp ffff88048a9e9c00, cur 1538882161 expire 1538882011 last 1538881959 Oct 06 20:20:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 06 20:20:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 747f1c2c-7790-f7f7-f917-37c59f8305f4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edfb800, cur 1538882442 expire 1538882292 last 1538882215 Oct 06 20:20:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 06 20:40:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a0cbb64-73cf-8a7a-8fbb-52e091912fb7 (at 10.8.11.36@o2ib6) Oct 06 20:41:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f181bc36-12f4-d13f-cac6-685c8fca4703 (at 10.8.11.21@o2ib6) Oct 06 20:41:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f4369c3f-1ca5-b9c7-e92e-2b81a126c246 (at 10.8.11.20@o2ib6) Oct 06 21:08:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e70ace12-bdc6-9f62-bdae-e4ad2e5904b0 (at 10.8.21.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e19c42400, cur 1538885326 expire 1538885176 last 1538885099 Oct 06 21:27:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9e0951a-b1a4-67a8-3107-4f9b1605d52b (at 10.8.21.33@o2ib6) Oct 06 21:40:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90f302c6-3e15-4da9-4efb-b6884037ab84 (at 10.9.106.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812383b3000, cur 1538887209 expire 1538887059 last 1538886982 Oct 06 22:03:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0d190998-2404-1e80-3f91-4622bd309ce3 (at 10.9.106.32@o2ib4) Oct 06 22:05:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5554029f-0e3e-042e-4f97-8165d6e7c7b7 (at 10.8.26.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0fd400, cur 1538888734 expire 1538888584 last 1538888507 Oct 06 22:24:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5554029f-0e3e-042e-4f97-8165d6e7c7b7 (at 10.8.26.24@o2ib6) Oct 06 22:24:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ff1edef3-0f3f-96c4-8ec4-14e922d3b712 (at 10.8.20.17@o2ib6) Oct 06 23:20:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 467f6573-776b-eb08-20fe-52cbf7495d50 (at 10.9.103.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be00400, cur 1538893231 expire 1538893081 last 1538893004 Oct 06 23:20:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 06 23:40:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3f3c223-b8c0-c770-b315-ba2fe2aa243f (at 10.9.101.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bee61c400, cur 1538894456 expire 1538894306 last 1538894229 Oct 06 23:44:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 467f6573-776b-eb08-20fe-52cbf7495d50 (at 10.9.103.12@o2ib4) Oct 07 00:02:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 25792081-1fe7-69fa-263d-f69775256ab2 (at 10.8.3.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803352b6400, cur 1538895723 expire 1538895573 last 1538895496 Oct 07 00:02:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbb83184-3e78-df7b-5109-069e289706b1 (at 10.9.101.54@o2ib4) Oct 07 00:14:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 585d8ff8-4575-4ba4-5af4-02aa57043521 (at 10.8.3.2@o2ib6) Oct 07 00:21:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c9ace87e-881e-0f5b-91c5-ab11f8e8ab54 (at 10.8.30.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c64f000, cur 1538896891 expire 1538896741 last 1538896664 Oct 07 00:38:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c9ace87e-881e-0f5b-91c5-ab11f8e8ab54 (at 10.8.30.30@o2ib6) Oct 07 00:42:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bd93288-61b3-614d-1b76-0b3ab3f8071d (at 10.9.101.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0f8400, cur 1538898150 expire 1538898000 last 1538897923 Oct 07 00:45:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da0a3ae4-6c54-3f50-b9f4-f32e0a8f3f64 (at 10.8.4.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a7c00, cur 1538898331 expire 1538898181 last 1538898104 Oct 07 00:45:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ddfd0ee-8c85-e8c0-a934-802dffd88125 (at 10.8.4.17@o2ib6) Oct 07 00:45:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09299d5d-d5ed-d979-4e97-6a3fa86ae647 (at 10.8.4.8@o2ib6) Oct 07 00:57:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa1d9318-3614-d909-899b-49b7c98fb443 (at 10.8.2.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ac06f8400, cur 1538899045 expire 1538898895 last 1538898818 Oct 07 00:57:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 00:57:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1585ed96-ba3a-af1b-3b88-056c0a1fcdb8 (at 10.8.2.2@o2ib6) Oct 07 01:04:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8b72551c-39d5-39a5-98c0-2e16e5031d3e (at 10.9.101.20@o2ib4) Oct 07 01:14:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6ca9875e-fe42-d597-4b85-6d682e442203 (at 10.8.3.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807d4eec800, cur 1538900089 expire 1538899939 last 1538899862 Oct 07 01:14:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 01:16:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 09ef7607-40ba-9fa7-233a-8ceca662a2c1 (at 10.9.101.26@o2ib4) in 162 seconds. I think it's dead, and I am evicting it. exp ffff881dfc5b6400, cur 1538900165 expire 1538900015 last 1538900003 Oct 07 01:16:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 01:20:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 702cf118-d587-206a-208a-9ed3f707a5d0 (at 10.9.101.9@o2ib4) Oct 07 01:31:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f4e86e85-f54f-00e9-0d8f-bb1ecc03b43e (at 10.8.3.6@o2ib6) Oct 07 01:39:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 23fcfacf-d0bc-678e-64c4-8bad17b50271 (at 10.9.101.26@o2ib4) Oct 07 01:39:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 84d21abe-f653-c4b0-019f-d33effb41b81 (at 10.9.101.10@o2ib4) Oct 07 02:06:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0bd9b61-def7-66ef-641a-9c6b884cf6af (at 10.8.21.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200ea65400, cur 1538903203 expire 1538903053 last 1538902976 Oct 07 02:18:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90317b30-96c6-5efa-b4ef-ab7c7138802e (at 10.9.101.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b1360b000, cur 1538903896 expire 1538903746 last 1538903669 Oct 07 02:19:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3647eeec-0059-88af-390d-1c392846296d (at 10.9.101.41@o2ib4) in 170 seconds. I think it's dead, and I am evicting it. exp ffff880ef8780c00, cur 1538903972 expire 1538903822 last 1538903802 Oct 07 02:19:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 07 02:24:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e469340-2578-0ab3-bd12-40d725255894 (at 10.9.101.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804bc21ec00, cur 1538904252 expire 1538904102 last 1538904025 Oct 07 02:24:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 02:25:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c7402119-795b-2902-b485-faa64d2953aa (at 10.8.21.17@o2ib6) Oct 07 02:29:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client faebe4f2-72fe-f0f9-1e2e-6103dcc6bcef (at 10.9.101.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af0000, cur 1538904571 expire 1538904421 last 1538904344 Oct 07 02:32:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d2b2375a-bb0a-88c5-0347-955f7d549254 (at 10.9.101.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc6b800, cur 1538904728 expire 1538904578 last 1538904501 Oct 07 02:35:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d66470e-e905-bc92-271f-83633f40b889 (at 10.9.101.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186c671000, cur 1538904954 expire 1538904804 last 1538904727 Oct 07 02:40:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14b712f1-d538-732a-e3dd-2e88d765d536 (at 10.9.101.42@o2ib4) Oct 07 02:41:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f3ec79b2-4842-1d7f-7e69-8775f4253bb2 (at 10.9.101.45@o2ib4) Oct 07 02:42:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1409c7c5-ea26-a0d0-4e2c-233fc67ef1fa (at 10.9.101.48@o2ib4) Oct 07 02:42:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6fe5ed74-50cb-72b3-2447-0b5a4d58c709 (at 10.9.101.56@o2ib4) Oct 07 02:42:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 02:45:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 86a8a040-246f-39c3-c30b-de653488110b (at 10.9.101.5@o2ib4) Oct 07 02:50:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 40d03a30-0722-73fb-2e85-6c7fa4fcb884 (at 10.9.101.12@o2ib4) Oct 07 02:55:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to efdeba46-8c0d-ccbe-eac1-6ff8fd5378d2 (at 10.9.101.31@o2ib4) Oct 07 02:59:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4503deeb-dc28-c96a-e9c5-e0039aae39c7 (at 10.9.101.11@o2ib4) Oct 07 03:00:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e0ed24a-77cf-5845-8414-6b4e824e694b (at 10.9.101.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fb5c2c00, cur 1538906446 expire 1538906296 last 1538906219 Oct 07 03:10:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 62ff57b6-1fdb-c8a0-ca56-f442ddf47f76 (at 10.9.101.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85a8b800, cur 1538907057 expire 1538906907 last 1538906830 Oct 07 03:12:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e3232aec-e7f7-7e53-e450-5748efbb099a (at 10.9.101.35@o2ib4) in 156 seconds. I think it's dead, and I am evicting it. exp ffff8818be6ae000, cur 1538907133 expire 1538906983 last 1538906977 Oct 07 03:15:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 285eb6c7-db3b-43cc-455e-eef716ebb40e (at 10.9.101.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe1345000, cur 1538907305 expire 1538907155 last 1538907078 Oct 07 03:18:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ce0f4a2-bbe5-c8ca-479a-5fade3452fe8 (at 10.9.101.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88037343a000, cur 1538907503 expire 1538907353 last 1538907276 Oct 07 03:24:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b7ccdc5f-cc21-8191-c7e2-4bad412b4af6 (at 10.9.101.17@o2ib4) Oct 07 03:34:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 62ff57b6-1fdb-c8a0-ca56-f442ddf47f76 (at 10.9.101.19@o2ib4) Oct 07 03:34:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5c6cb92-7c6a-bc0a-275a-4a9a89f72ec8 (at 10.9.101.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014e258800, cur 1538908481 expire 1538908331 last 1538908254 Oct 07 03:35:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dbbde0f5-0421-507a-5b2e-070c7b9ba1d8 (at 10.9.101.35@o2ib4) Oct 07 03:37:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 03be79dc-6843-d91f-4518-15178cd82d7b (at 10.9.101.21@o2ib4) Oct 07 03:39:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 74957cfa-58d3-37c5-2dfa-290aa89dcd30 (at 10.9.101.14@o2ib4) Oct 07 03:50:39 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 07 03:56:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ec18324c-6eaa-7fe8-1c48-55ed93048960 (at 10.9.101.50@o2ib4) Oct 07 04:38:34 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 07 04:38:34 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 07 04:48:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bddc3ae-ca6a-6b7f-9271-db2607a189cf (at 10.9.101.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef107000, cur 1538912928 expire 1538912778 last 1538912701 Oct 07 05:10:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 55d74724-b2a5-19c3-b749-ab2658f18863 (at 10.9.101.43@o2ib4) Oct 07 05:29:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f38fbd6-9db3-b255-018c-2ebb11dd1643 (at 10.8.20.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88010988c800, cur 1538915366 expire 1538915216 last 1538915139 Oct 07 05:30:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a46f3f2b-9d47-1223-c851-b5657b5b0643 (at 10.9.101.44@o2ib4) in 188 seconds. I think it's dead, and I am evicting it. exp ffff881e415b5c00, cur 1538915442 expire 1538915292 last 1538915254 Oct 07 05:36:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e18e66d3-ff2c-ccce-6ce5-71d956844e5c (at 10.8.4.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c7fa21400, cur 1538915788 expire 1538915638 last 1538915561 Oct 07 05:47:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5f38fbd6-9db3-b255-018c-2ebb11dd1643 (at 10.8.20.27@o2ib6) Oct 07 05:53:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 583f3ef7-dfbf-e305-3ba8-2db1fc44b331 (at 10.9.101.44@o2ib4) Oct 07 06:01:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dafb6a59-e11b-03ee-af38-b4b95700f3c7 (at 10.8.4.22@o2ib6) Oct 07 06:16:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c4ac6b9-aa38-89a3-e4bf-8aaef9ad0079 (at 10.8.24.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e46400, cur 1538918190 expire 1538918040 last 1538917963 Oct 07 06:34:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9c4ac6b9-aa38-89a3-e4bf-8aaef9ad0079 (at 10.8.24.35@o2ib6) Oct 07 06:49:08 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 07 06:56:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 77ef26e8-3f2f-426f-373b-6a2b8dbf05d0 (at 10.9.101.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807444e9c00, cur 1538920577 expire 1538920427 last 1538920350 Oct 07 06:59:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d78e855d-2c60-55bf-8431-ec4c336e5ad1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf93ec00, cur 1538920746 expire 1538920596 last 1538920519 Oct 07 07:01:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 07 07:07:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 24a2f7ba-ac48-9109-fe82-8df085bcebe7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3b9800, cur 1538921233 expire 1538921083 last 1538921006 Oct 07 07:07:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 07 07:11:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 07 07:11:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04557847-3270-ecd0-2465-b27454880bf6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe721c400, cur 1538921510 expire 1538921360 last 1538921283 Oct 07 07:16:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8cdf4b2b-182f-792c-365f-a0ab06f47ac5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc080d400, cur 1538921781 expire 1538921631 last 1538921554 Oct 07 07:16:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 07:17:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43241e91-cff0-437e-bacc-48942c957e6a (at 10.9.101.51@o2ib4) Oct 07 07:30:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1efa7ead-55ab-c60c-2c5f-9fc60d5c738a (at 10.8.24.20@o2ib6) Oct 07 07:33:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aed94a20-49cc-738d-c19c-035f7d7c4c7f (at 10.8.24.15@o2ib6) Oct 07 08:19:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f81d344c-376a-9dfa-8943-347a6a125f20 (at 10.9.105.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc68400, cur 1538925576 expire 1538925426 last 1538925349 Oct 07 08:19:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 08:25:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55df590f-6d35-ebd7-4f06-a6fc115c41da (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e19c41000, cur 1538925954 expire 1538925804 last 1538925727 Oct 07 08:42:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d864271d-68ba-37ce-9609-2c6659593006 (at 10.9.105.28@o2ib4) Oct 07 08:44:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 08:51:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 08:51:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5bf6e02-53e0-aa24-9419-5a2de3026023 (at 10.8.22.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0aa400, cur 1538927497 expire 1538927347 last 1538927270 Oct 07 08:52:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3f0fab7e-a359-b302-6fee-0862487323a5 (at 10.8.24.30@o2ib6) in 187 seconds. I think it's dead, and I am evicting it. exp ffff880ee604cc00, cur 1538927573 expire 1538927423 last 1538927386 Oct 07 08:52:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 08:59:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2da4c1b8-e7b3-7270-75b4-480a192bd215 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb81800, cur 1538927986 expire 1538927836 last 1538927759 Oct 07 09:01:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 09:06:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9977b783-8bae-c617-571d-8e99aecdae44 (at 10.8.20.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4d000, cur 1538928378 expire 1538928228 last 1538928151 Oct 07 09:10:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dda06d63-db84-06c7-6cb3-bf6f78259ebf (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88152c643800, cur 1538928606 expire 1538928456 last 1538928379 Oct 07 09:12:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3f0fab7e-a359-b302-6fee-0862487323a5 (at 10.8.24.30@o2ib6) Oct 07 09:12:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 09:14:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a31135bd-8bcc-7242-cf2c-3dc969b0afda (at 10.8.22.6@o2ib6) Oct 07 09:14:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d5f16394-595b-8516-4113-536764c05b5b (at 10.8.20.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e2c00, cur 1538928891 expire 1538928741 last 1538928664 Oct 07 09:20:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bfc31d28-08e5-149e-56c7-09e169b4b5c2 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881846f5fc00, cur 1538929227 expire 1538929077 last 1538929000 Oct 07 09:21:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 09:24:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9977b783-8bae-c617-571d-8e99aecdae44 (at 10.8.20.14@o2ib6) Oct 07 09:27:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20a7716b-26ff-2069-8765-1591ef050c3c (at 10.8.26.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881eccecf000, cur 1538929653 expire 1538929503 last 1538929426 Oct 07 09:32:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8b65623d-6d36-c44f-e2f5-57d63c3b98c8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7de6c400, cur 1538929929 expire 1538929779 last 1538929702 Oct 07 09:33:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 09:35:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c900a5b6-e9f2-a9ae-6db6-9e1bc7d44aac (at 10.9.105.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881858ac1000, cur 1538930112 expire 1538929962 last 1538929885 Oct 07 09:36:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d5f16394-595b-8516-4113-536764c05b5b (at 10.8.20.30@o2ib6) Oct 07 09:42:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cbe7dfe8-8372-2693-42f0-4c93db73dfca (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d0e236800, cur 1538930520 expire 1538930370 last 1538930293 Oct 07 09:43:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 09:47:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 20a7716b-26ff-2069-8765-1591ef050c3c (at 10.8.26.5@o2ib6) Oct 07 09:54:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0c7eefa3-fc99-540c-06e7-6b75b4a7596a (at 10.12.4.71@o2ib) Oct 07 10:33:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7039c8b5-a301-5580-e07f-95e2f8f5128e (at 10.8.30.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edfc800, cur 1538933639 expire 1538933489 last 1538933412 Oct 07 10:38:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 86211b14-93ed-feb3-78f9-d5cc0479d06c (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880036e51000, cur 1538933912 expire 1538933762 last 1538933685 Oct 07 10:41:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 10:41:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 10:42:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28a85898-d85d-64e1-3b8c-055ea3d793a5 (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039231800, cur 1538934142 expire 1538933992 last 1538933915 Oct 07 10:47:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 10:48:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c4978989-7e7d-58e8-c379-856720cae211 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef28dcc00, cur 1538934484 expire 1538934334 last 1538934257 Oct 07 10:50:06 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 07 10:50:06 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 07 10:51:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 58382bcb-2eee-c6e0-32f0-dc777b7e1ee7 (at 10.8.30.16@o2ib6) Oct 07 10:58:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ecc1d83-b2d0-3a61-325a-167cdc30bb55 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88094e959c00, cur 1538935098 expire 1538934948 last 1538934871 Oct 07 10:59:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 07 11:31:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a659c13b-db21-f6df-12e2-e1c3ec26d6b5 (at 10.8.30.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda289000, cur 1538937102 expire 1538936952 last 1538936875 Oct 07 11:39:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 152c2466-b73e-3ea0-eb62-f44791d9a8fe (at 10.8.30.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee6048000, cur 1538937553 expire 1538937403 last 1538937326 Oct 07 11:50:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 65830a19-b43c-97ae-7b09-175058c60244 (at 10.8.30.15@o2ib6) Oct 07 11:53:27 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 07 11:57:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 25307525-84e2-b7a3-1318-3963f6f01bb5 (at 10.8.30.12@o2ib6) Oct 07 12:23:41 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 07 12:41:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 431a2522-6672-de37-43ad-f34ec96f73cb (at 10.8.24.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea512a800, cur 1538941303 expire 1538941153 last 1538941076 Oct 07 13:00:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 838b6aba-d86f-da47-f02f-ebbbd61a0456 (at 10.8.24.6@o2ib6) Oct 07 13:28:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9cc5c9d7-9e45-345c-5466-c70d958002f8 (at 10.8.6.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a5f77bc00, cur 1538944099 expire 1538943949 last 1538943872 Oct 07 13:29:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5b98cccf-6591-3761-1ccb-22ff0c718b98 (at 10.8.12.10@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff88014c050c00, cur 1538944175 expire 1538944025 last 1538944004 Oct 07 13:54:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1ceb6364-742e-156f-1f57-f0cfb2eb59b7 (at 10.8.12.10@o2ib6) Oct 07 13:59:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Oct 07 14:06:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8fd52fde-462c-b60f-3af5-ba7330b6346f (at 10.8.23.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c64ec00, cur 1538946389 expire 1538946239 last 1538946162 Oct 07 14:15:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4d976fdb-6c00-f2f8-698c-96398c3acf52 (at 10.8.21.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881700399c00, cur 1538946929 expire 1538946779 last 1538946702 Oct 07 14:15:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 14:23:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 212f1f1d-e92a-8535-4e55-ecd683037895 (at 10.8.23.1@o2ib6) Oct 07 14:27:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8fd52fde-462c-b60f-3af5-ba7330b6346f (at 10.8.23.11@o2ib6) Oct 07 14:32:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7b7c304b-0418-5474-49ca-ead1c89864ad (at 10.8.21.28@o2ib6) Oct 07 14:39:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43f496f6-cadc-409f-836c-46c2a9950d52 (at 10.8.23.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200ea67c00, cur 1538948399 expire 1538948249 last 1538948172 Oct 07 14:57:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 096d7c4d-4f27-b76a-aa88-24fe1294a901 (at 10.8.9.2@o2ib6) Oct 07 14:58:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43f496f6-cadc-409f-836c-46c2a9950d52 (at 10.8.23.29@o2ib6) Oct 07 14:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cadecfb3-b2c7-76db-7dea-08467922e62b (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d39b4b400, cur 1538949562 expire 1538949412 last 1538949335 Oct 07 15:16:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 525f047d-a661-6241-c6b8-cff31c5c0766 (at 10.9.102.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565cd400, cur 1538950607 expire 1538950457 last 1538950380 Oct 07 15:30:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c72ffe48-b4c7-8782-cfe6-5bb308707a9e (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f56b11000, cur 1538951408 expire 1538951258 last 1538951181 Oct 07 15:30:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 15:36:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49b5291e-7bed-bbdc-be10-338a44959428 (at 10.8.6.26@o2ib6) Oct 07 15:39:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 525f047d-a661-6241-c6b8-cff31c5c0766 (at 10.9.102.64@o2ib4) Oct 07 15:40:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0225c4f2-f9fe-b03e-b5ee-2a3c4da0c534 (at 10.9.102.63@o2ib4) Oct 07 15:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7201811-2ee1-f16c-598e-18367f2338d7 (at 10.12.4.33@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f82400, cur 1538953162 expire 1538953012 last 1538952935 Oct 07 15:59:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 16:12:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 790c6f8f-26ec-41ff-6eef-2fa119576e77 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afec00, cur 1538953926 expire 1538953776 last 1538953699 Oct 07 16:30:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 790c6f8f-26ec-41ff-6eef-2fa119576e77 (at 10.8.26.4@o2ib6) Oct 07 17:47:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc688f68-d18a-bcca-dfdd-a201f1e99eaa (at 10.8.22.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef6800, cur 1538959651 expire 1538959501 last 1538959424 Oct 07 18:06:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bc688f68-d18a-bcca-dfdd-a201f1e99eaa (at 10.8.22.34@o2ib6) Oct 07 18:12:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4292cbb5-9c8d-3fef-1d39-6d67f5cd582b (at 10.8.31.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3f3400, cur 1538961126 expire 1538960976 last 1538960899 Oct 07 18:30:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4292cbb5-9c8d-3fef-1d39-6d67f5cd582b (at 10.8.31.4@o2ib6) Oct 07 18:47:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be1a52a8-b504-e2a2-6f55-fcb79b0f2990 (at 10.8.2.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4a400, cur 1538963242 expire 1538963092 last 1538963015 Oct 07 18:51:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5bf085c4-7698-cb73-d91c-d388aeb8271a (at 10.8.2.14@o2ib6) Oct 07 19:52:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 25aeada7-ff11-3215-8cc7-0614d150fe0b (at 10.8.26.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808d6af0c00, cur 1538967142 expire 1538966992 last 1538966915 Oct 07 20:11:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 111517b6-df87-c3f3-4f70-e224edfc9255 (at 10.8.26.31@o2ib6) Oct 07 20:11:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 95d3d72c-8035-09f5-a5ad-3dea6e11befd (at 10.8.26.29@o2ib6) Oct 07 20:13:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d43a0494-5b01-8a60-bf09-5b889bc08dc4 (at 10.8.21.24@o2ib6) Oct 07 21:27:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b42f6f82-e815-cc46-9fbe-2b7e72ce3335 (at 10.8.10.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88170039cc00, cur 1538972867 expire 1538972717 last 1538972640 Oct 07 21:27:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 07 21:51:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b42f6f82-e815-cc46-9fbe-2b7e72ce3335 (at 10.8.10.2@o2ib6) Oct 07 23:28:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 86fd3027-f92e-81f0-440a-9f5a330819a0 (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28a400, cur 1538980100 expire 1538979950 last 1538979873 Oct 07 23:31:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a524ba4-15c7-1347-2921-cd73173ea987 (at 10.12.4.33@o2ib) Oct 07 23:31:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0c7eefa3-fc99-540c-06e7-6b75b4a7596a (at 10.12.4.71@o2ib) Oct 07 23:40:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f10b3957-3abe-811c-2247-b8da1e60ecc7 (at 10.8.26.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803352b0400, cur 1538980812 expire 1538980662 last 1538980585 Oct 07 23:45:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b809926d-5c68-cfa7-8499-6efb5a5e687e (at 10.8.30.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014e25d400, cur 1538981152 expire 1538981002 last 1538980925 Oct 07 23:45:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 07 23:46:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09db04e9-acc9-b441-3d92-e291dda8f0be (at 10.8.30.17@o2ib6) Oct 07 23:52:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 80f4a5fe-0b34-4971-9a03-0957e0634b60 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d88957800, cur 1538981531 expire 1538981381 last 1538981304 Oct 07 23:53:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Oct 07 23:57:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 92c1ca93-68ed-d713-b841-a6e421606f1a (at 10.8.26.1@o2ib6) Oct 07 23:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f4ab7c-96d6-a80d-e451-2fe219bc7544 (at 10.8.30.10@o2ib6) Oct 08 00:04:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f2576ed-8160-84a8-4db1-8cb59d8dd998 (at 10.8.24.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809439dd400, cur 1538982252 expire 1538982102 last 1538982025 Oct 08 00:04:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b809926d-5c68-cfa7-8499-6efb5a5e687e (at 10.8.30.25@o2ib6) Oct 08 00:21:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 93e10cbd-3f43-1956-e751-603800eb73db (at 10.8.24.31@o2ib6) Oct 08 00:46:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bbb229d1-f487-bbde-624d-04748ce8928c (at 10.8.24.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dda818c00, cur 1538984809 expire 1538984659 last 1538984582 Oct 08 00:57:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49436388-6e40-0ef9-1610-3273dbbd6cd8 (at 10.8.4.10@o2ib6) Oct 08 00:57:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 36b3bfc7-995f-43bd-9c82-6af6cded28cb (at 10.8.4.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ac06ffc00, cur 1538985470 expire 1538985320 last 1538985243 Oct 08 01:05:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 702be5ee-77fd-a649-ec5e-0b8c1a5a72e3 (at 10.8.24.33@o2ib6) Oct 08 01:26:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ccb87cd1-4353-8ff4-a8e7-89396c8dc9f9 (at 10.8.22.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea512d400, cur 1538987177 expire 1538987027 last 1538986950 Oct 08 01:36:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 68f50046-7267-6faa-59c3-d23ced95d00c (at 10.8.20.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88094e958400, cur 1538987774 expire 1538987624 last 1538987547 Oct 08 01:43:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6465767f-c9b2-e741-68ff-a04f7be36164 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101cff7000, cur 1538988205 expire 1538988055 last 1538987978 Oct 08 01:44:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ccb87cd1-4353-8ff4-a8e7-89396c8dc9f9 (at 10.8.22.28@o2ib6) Oct 08 01:50:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Oct 08 03:15:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cee97693-3e96-56b6-42e3-eac893935712 (at 10.9.103.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882037d76800, cur 1538993724 expire 1538993574 last 1538993497 Oct 08 03:34:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52cd6d8a-be9c-aba0-3385-b2bd5b00425d (at 10.8.24.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3aff000, cur 1538994898 expire 1538994748 last 1538994671 Oct 08 03:40:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cee97693-3e96-56b6-42e3-eac893935712 (at 10.9.103.23@o2ib4) Oct 08 03:53:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3a8abb12-6d36-ed5e-c2e9-6b91facfbebc (at 10.8.24.25@o2ib6) Oct 08 04:10:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f955755-d391-66d1-f722-6ec6b0490b32 (at 10.9.103.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565ca400, cur 1538997031 expire 1538996881 last 1538996804 Oct 08 04:25:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ee4ed4ca-d780-30b6-dcfb-56cbb1a58d08 (at 10.8.10.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b29bff800, cur 1538997939 expire 1538997789 last 1538997712 Oct 08 04:34:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9f955755-d391-66d1-f722-6ec6b0490b32 (at 10.9.103.15@o2ib4) Oct 08 04:49:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5f847b31-ead3-0f1f-ad35-9c262bc96b44 (at 10.8.10.34@o2ib6) Oct 08 05:07:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4dbe04f-9a4c-d6a6-747d-697a5eeabf0a (at 10.9.103.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103af65800, cur 1539000459 expire 1539000309 last 1539000232 Oct 08 05:30:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f4dbe04f-9a4c-d6a6-747d-697a5eeabf0a (at 10.9.103.21@o2ib4) Oct 08 05:30:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d43a0494-5b01-8a60-bf09-5b889bc08dc4 (at 10.8.21.24@o2ib6) Oct 08 05:31:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1deeb823-2c77-d3a4-a5c7-75c27b439f74 (at 10.8.21.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880727400000, cur 1539001914 expire 1539001764 last 1539001687 Oct 08 05:43:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47087cf4-efb8-ad27-8595-82bcf1e371cc (at 10.8.12.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185d7d6800, cur 1539002638 expire 1539002488 last 1539002411 Oct 08 05:55:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a62362ed-4ae7-ae15-6f0c-2e3f13b1af84 (at 10.8.10.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185d7d3000, cur 1539003307 expire 1539003157 last 1539003080 Oct 08 06:07:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f5a1a98a-f3c9-c19e-83f9-874ab3c48e89 (at 10.8.12.3@o2ib6) Oct 08 06:16:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3f258832-551a-9af0-1363-7bbad31ea67a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ce878ac00, cur 1539004576 expire 1539004426 last 1539004349 Oct 08 06:16:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 06:19:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 598172c7-882b-bb28-df75-70a135b42b3a (at 10.8.10.20@o2ib6) Oct 08 07:21:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 07:21:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 16c857ed-4e2c-8dcd-8b74-68c6d8fe6c9c (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7b12400, cur 1539008511 expire 1539008361 last 1539008284 Oct 08 08:00:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 654ae3a6-9131-6554-1f85-b999c69aef36 (at 10.8.11.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3f2800, cur 1539010854 expire 1539010704 last 1539010627 Oct 08 08:16:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 08:17:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 975ec419-dc8f-4c75-cea5-7638e3f425e1 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a30f400, cur 1539011853 expire 1539011703 last 1539011626 Oct 08 08:22:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 08:23:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0dac2916-3ae7-dd2a-1658-031195d760f8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f56b12000, cur 1539012229 expire 1539012079 last 1539012002 Oct 08 08:25:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 035f7e72-5f28-c5ed-1520-89b77f2bda76 (at 10.8.11.25@o2ib6) Oct 08 08:26:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 08 08:27:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b8cacac-9c14-2d13-a647-d032adb265e6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd65e400, cur 1539012454 expire 1539012304 last 1539012227 Oct 08 08:41:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c3e98c37-6c62-f76f-b4c5-de4c69d99884 (at 10.9.112.14@o2ib4) Oct 08 08:41:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Oct 08 08:43:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Oct 08 08:43:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to da81cd8a-54bb-1bcc-ecce-162331019b10 (at 10.9.103.5@o2ib4) Oct 08 08:45:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7fb608ea-7572-da85-14d9-102571330c30 (at 10.9.103.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc71d9800, cur 1539013530 expire 1539013380 last 1539013303 Oct 08 08:46:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 08 08:48:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ad68e508-e4d1-4df9-27a9-b38e1beb0fd1 (at 10.8.27.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8c875800, cur 1539013697 expire 1539013547 last 1539013470 Oct 08 08:53:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e76d7a93-55b4-1e86-6751-3a9e278ca5ba (at 10.9.101.57@o2ib4) Oct 08 08:54:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9facfd8-b4d3-8631-92cb-cd04680239e7 (at 10.9.101.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a57400, cur 1539014077 expire 1539013927 last 1539013850 Oct 08 08:54:37 oak-md1-s2 kernel: Lustre: Skipped 18 previous similar messages Oct 08 08:54:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Oct 08 08:58:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e58ffc3b-fe0c-3954-fbfb-531913b2191d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815bc237000, cur 1539014297 expire 1539014147 last 1539014070 Oct 08 08:59:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fdd03f0c-bdec-7fc5-d7e8-f97eab72c990 (at 10.9.0.64@o2ib4) in 215 seconds. I think it's dead, and I am evicting it. exp ffff8817eaf03800, cur 1539014373 expire 1539014223 last 1539014158 Oct 08 08:59:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 09:11:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Oct 08 09:12:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ff851e07-ff5a-cfc6-2e8f-89baeb78be9a (at 10.8.27.19@o2ib6) Oct 08 09:12:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 09:12:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c0ba1883-d65e-efa3-7adb-ba8d7a828d36 (at 10.8.27.17@o2ib6) Oct 08 09:13:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 119f7486-6b19-c459-64bb-c376586d7bf0 (at 10.8.24.22@o2ib6) Oct 08 09:13:18 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 09:16:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bf0fb52f-d283-67ab-2acb-580c45fa616c (at 10.9.101.4@o2ib4) Oct 08 09:16:47 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 09:20:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d342f90e-597c-3ec6-8e55-42583e98e20d (at 10.9.107.31@o2ib4) Oct 08 09:20:05 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 09:26:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2b8fec05-df54-0626-776b-d9c2a1a69b47 (at 10.9.0.63@o2ib4) Oct 08 09:26:20 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 08 09:33:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2724bbcd-af4e-1677-0bd5-b9dbc491346e (at 10.8.22.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d32eea400, cur 1539016398 expire 1539016248 last 1539016171 Oct 08 09:40:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3bb33aa5-dd6c-65c8-8d51-624395f29422 (at 10.9.114.8@o2ib4) Oct 08 09:40:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 09:43:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20f17997-e9f4-faa0-4016-2c70dfcc840a (at 10.8.21.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab3ef2c00, cur 1539017005 expire 1539016855 last 1539016778 Oct 08 09:43:41 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539017014/real 1539017014] req@ffff881cb0b75700 x1610631035590880/t0(0) o104->oak-MDT0000@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539017021 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 08 09:43:41 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Oct 08 09:43:48 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539017021/real 1539017021] req@ffff881cb0b75700 x1610631035590880/t0(0) o104->oak-MDT0000@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539017028 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 08 09:44:02 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539017035/real 1539017035] req@ffff881cb0b75700 x1610631035590880/t0(0) o104->oak-MDT0000@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539017042 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 08 09:44:02 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 08 09:44:24 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539017056/real 1539017056] req@ffff881cb0b75700 x1610631035590880/t0(0) o104->oak-MDT0000@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539017063 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 08 09:44:24 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 08 09:44:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0f64e671-a2de-1953-48be-3fdb8d7c74f3 (at 10.8.30.20@o2ib6) in 224 seconds. I think it's dead, and I am evicting it. exp ffff881e25d21400, cur 1539017081 expire 1539016931 last 1539016857 Oct 08 09:44:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 09:45:06 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539017099/real 1539017099] req@ffff881cb0b75700 x1610631035590880/t0(0) o104->oak-MDT0000@10.8.0.65@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539017106 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 08 09:45:06 oak-md1-s2 kernel: Lustre: 10666:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 08 09:45:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dcefb1ad-9947-7dde-f2cb-d5733ee0ba68 (at 10.8.0.66@o2ib6) in 182 seconds. I think it's dead, and I am evicting it. exp ffff880fe7fa1400, cur 1539017157 expire 1539017007 last 1539016975 Oct 08 09:45:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 09:51:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f066dded-f6ec-4818-731a-b0e1cfb98fd3 (at 10.8.22.3@o2ib6) Oct 08 09:51:46 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 09:55:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37cbc4e6-831b-d2dc-c42c-4d57d874c5b7 (at 10.8.16.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d24800, cur 1539017752 expire 1539017602 last 1539017525 Oct 08 09:55:52 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 10:01:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa459b95-7adc-018e-6715-5d33e2a43de4 (at 10.8.30.19@o2ib6) Oct 08 10:01:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 10:12:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 08 10:12:52 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 08 10:23:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 08 10:23:02 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 08 10:30:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c18d7cf-95f0-f9af-9969-5890adf67855 (at 10.9.103.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b7db400, cur 1539019825 expire 1539019675 last 1539019598 Oct 08 10:30:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 10:39:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2b8fec05-df54-0626-776b-d9c2a1a69b47 (at 10.9.0.63@o2ib4) Oct 08 10:39:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 10:54:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e62e7533-1106-7bae-e0ae-cddffbed489e (at 10.8.23.31@o2ib6) Oct 08 10:54:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 12:03:19 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 12:06:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 213a9820-6652-e195-499f-76a8202d982b (at 10.8.22.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4b800, cur 1539025579 expire 1539025429 last 1539025352 Oct 08 12:27:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 213a9820-6652-e195-499f-76a8202d982b (at 10.8.22.15@o2ib6) Oct 08 12:27:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 12:29:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d473ca75-37ef-d163-f7a7-9bc458adb23e (at 10.8.20.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804b9f8ac00, cur 1539026970 expire 1539026820 last 1539026743 Oct 08 12:36:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33a538d3-6413-2759-257c-1c1e638a0f1f (at 10.12.4.22@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da97ecc00, cur 1539027419 expire 1539027269 last 1539027192 Oct 08 12:47:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 08 12:48:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client adc9c420-f8c2-11dd-abd7-23e8a2d5f580 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88085dc1e000, cur 1539028081 expire 1539027931 last 1539027854 Oct 08 12:50:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d473ca75-37ef-d163-f7a7-9bc458adb23e (at 10.8.20.33@o2ib6) Oct 08 13:50:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 986bf173-8a48-8326-0b87-c02bc7db3b53 (at 10.8.9.8@o2ib6) Oct 08 13:51:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1cf0aab1-c055-080d-76e5-bb1d41541bdd (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0a6fe800, cur 1539031915 expire 1539031765 last 1539031688 Oct 08 13:53:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ddd994e0-54e3-2076-d815-d0b1403e734c (at 10.8.26.33@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff881843ed6400, cur 1539031991 expire 1539031841 last 1539031820 Oct 08 13:53:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 08 13:56:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 08 13:57:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b62f894d-b7ec-f071-7bd5-4112886fd970 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802da793800, cur 1539032251 expire 1539032101 last 1539032024 Oct 08 13:57:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 14:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb131794-f3a4-db0e-0813-4ac20448623b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c43be0400, cur 1539032520 expire 1539032370 last 1539032293 Oct 08 14:03:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 14:06:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ece8e43-3dfb-0b94-0b10-cd9628a4f30c (at 10.9.103.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef287ec00, cur 1539032778 expire 1539032628 last 1539032551 Oct 08 14:06:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 14:07:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ef53743-3f0e-10f5-c56a-ca04a37a64cf (at 10.8.20.19@o2ib6) in 210 seconds. I think it's dead, and I am evicting it. exp ffff880ba78c7c00, cur 1539032854 expire 1539032704 last 1539032644 Oct 08 14:09:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 08 14:10:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 12d614e9-6869-b138-6b33-0c034c8bcd62 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e46000, cur 1539033054 expire 1539032904 last 1539032827 Oct 08 14:10:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 14:16:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 14:16:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 14:16:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d04c1d2e-35ef-0c0c-59db-6d39dab71477 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ade32c400, cur 1539033410 expire 1539033260 last 1539033183 Oct 08 14:19:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 14:20:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20d1dc92-8899-f042-6e18-063df324b8cd (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ed255800, cur 1539033631 expire 1539033481 last 1539033404 Oct 08 14:23:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 13939d8d-6d1c-8779-98a6-789026df278e (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ed2221c00, cur 1539033836 expire 1539033686 last 1539033609 Oct 08 14:25:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 14:25:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 14:30:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b8c35819-c41a-ea5d-c1cc-0d63ca35ab7d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f59a43800, cur 1539034211 expire 1539034061 last 1539033984 Oct 08 14:30:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 14:36:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 14:36:07 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 14:40:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a8702b4b-bb63-bf28-aec3-3f0c1fa56985 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ee807400, cur 1539034827 expire 1539034677 last 1539034600 Oct 08 14:40:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 14:48:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 14:48:18 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 14:55:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4dcedbad-da6f-c011-a071-465687762e55 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c93ff4800, cur 1539035751 expire 1539035601 last 1539035524 Oct 08 14:55:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 15:01:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 15:01:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 15:09:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28cec1e7-4b0e-0f99-2fa2-9c3b0b2e10b1 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b3a3f800, cur 1539036547 expire 1539036397 last 1539036320 Oct 08 15:09:07 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 15:22:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e444a4f4-b999-183d-0054-30647a1e83a5 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880052c80800, cur 1539037329 expire 1539037179 last 1539037102 Oct 08 15:29:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 08 15:29:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 15:38:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to da81cd8a-54bb-1bcc-ecce-162331019b10 (at 10.9.103.5@o2ib4) Oct 08 15:39:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2bbfcecb-be61-f784-ec01-84d47b647d4a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88152167c800, cur 1539038374 expire 1539038224 last 1539038147 Oct 08 15:39:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 15:46:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Oct 08 15:50:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3f5ab6eb-d9da-7c5b-f303-6a9fa89e48d2 (at 10.9.103.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2115c00, cur 1539039016 expire 1539038866 last 1539038789 Oct 08 15:50:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 15:52:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 15:52:01 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 16:03:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 08 16:03:09 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 16:08:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37b209a1-7dc0-4482-fae9-23742f551ec0 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb85400, cur 1539040093 expire 1539039943 last 1539039866 Oct 08 16:08:13 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 08 16:15:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 08 16:15:29 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 08 16:28:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 16:28:02 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 16:28:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ea0d3cd-15ed-0a4f-eeb6-7405183920cb (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d1464dc00, cur 1539041315 expire 1539041165 last 1539041088 Oct 08 16:28:35 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 08 16:29:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0f95c06c-d1be-b2fc-5ede-eea969c41f88 (at 10.8.17.26@o2ib6) in 213 seconds. I think it's dead, and I am evicting it. exp ffff88068aa2a400, cur 1539041391 expire 1539041241 last 1539041178 Oct 08 16:34:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6743fb7-90ec-122d-0bb8-9815a5336df8 (at 10.8.22.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa013c00, cur 1539041669 expire 1539041519 last 1539041442 Oct 08 16:34:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 16:42:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 40de5074-fc43-21b5-7793-24d3541afd98 (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a98404000, cur 1539042134 expire 1539041984 last 1539041907 Oct 08 16:42:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 08 16:42:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 16:53:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6743fb7-90ec-122d-0bb8-9815a5336df8 (at 10.8.22.12@o2ib6) Oct 08 17:17:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 308315e8-7143-45d4-fdfa-b2fc027f768d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f96a51800, cur 1539044227 expire 1539044077 last 1539044000 Oct 08 17:17:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 17:17:25 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 17:21:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 17:22:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0f581583-08eb-789d-7211-c4949c5f2a3e (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881846f5f800, cur 1539044547 expire 1539044397 last 1539044320 Oct 08 17:26:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e8a11047-7093-7fad-5b18-0b1a2325936b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ade32ac00, cur 1539044808 expire 1539044658 last 1539044581 Oct 08 17:27:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 17:32:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b78852b-a7ed-98f7-0879-73b6ec09af19 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee1a02800, cur 1539045171 expire 1539045021 last 1539044944 Oct 08 17:34:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 08 17:34:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 17:44:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 17:44:58 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 08 17:45:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c76676cd-b273-f73e-d1fb-155bac04c50e (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e6400, cur 1539045925 expire 1539045775 last 1539045698 Oct 08 17:45:25 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 17:59:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a5373d38-12b4-d283-7495-27ac13bd25f6 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e6bd400, cur 1539046754 expire 1539046604 last 1539046527 Oct 08 17:59:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 18:03:59 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539047032/real 1539047032] req@ffff88177d16e600 x1610631080265312/t0(0) o104->oak-MDT0000@10.9.104.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1539047039 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 08 18:03:59 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Oct 08 18:04:13 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539047046/real 1539047046] req@ffff88177d16e600 x1610631080265312/t0(0) o104->oak-MDT0000@10.9.104.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1539047053 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 08 18:04:13 oak-md1-s2 kernel: Lustre: 10617:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 08 18:24:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 08 18:24:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 18:26:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 08 18:26:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 18:30:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e39a6744-58a7-ceb5-6ea2-3e1a82106b82 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88179c235400, cur 1539048621 expire 1539048471 last 1539048394 Oct 08 18:30:21 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 08 18:31:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e715dde-5515-b6e2-a7bd-45b19bd80714 (at 10.8.17.26@o2ib6) in 194 seconds. I think it's dead, and I am evicting it. exp ffff8805feaf8000, cur 1539048697 expire 1539048547 last 1539048503 Oct 08 18:31:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 08 18:40:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 18:58:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa97e8f3-5b88-8f68-8646-51f442730827 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88197aa5f800, cur 1539050318 expire 1539050168 last 1539050091 Oct 08 19:07:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 19:21:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 08 19:22:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a207908a-e6d5-68f2-9ef9-66a74f85450a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a7e872000, cur 1539051721 expire 1539051571 last 1539051494 Oct 08 19:24:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 19:25:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 838c9e52-9853-2ba1-0823-0a9647b565a6 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe5bdf000, cur 1539051950 expire 1539051800 last 1539051723 Oct 08 19:28:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 19:29:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2732aec-4742-e6f5-b922-c77b17fff4fc (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e3c400, cur 1539052197 expire 1539052047 last 1539051970 Oct 08 19:31:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 19:32:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02eba05e-2125-77d2-1f81-cf2639e19240 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810302f6800, cur 1539052366 expire 1539052216 last 1539052139 Oct 08 19:36:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 19:37:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9c8fcc8-7c56-9c5d-9206-445ab41a8cb6 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e95c9ac00, cur 1539052658 expire 1539052508 last 1539052431 Oct 08 19:43:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 19:44:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d3e488b-f511-8a88-150b-c5eaa3e858a5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880038e8bc00, cur 1539053087 expire 1539052937 last 1539052860 Oct 08 19:49:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc775e7c-38e1-5b91-0588-91e87b5d7e4a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881875ecec00, cur 1539053362 expire 1539053212 last 1539053135 Oct 08 20:13:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f903d23-af18-2e00-c921-e1ab5fc51037 (at 10.9.102.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba2000, cur 1539054822 expire 1539054672 last 1539054595 Oct 08 20:14:45 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 20:16:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 20:16:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 20:16:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 08f4da73-b22b-d5e7-c5e7-543087f739bd (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881816ee7400, cur 1539054994 expire 1539054844 last 1539054767 Oct 08 20:20:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 20:21:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04d03a38-1985-638f-a660-7a0d5986a359 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dfc5b4800, cur 1539055274 expire 1539055124 last 1539055047 Oct 08 20:28:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 08 20:29:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39d6ef44-4c3f-de55-feb3-fca9a1c653be (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809a6fef800, cur 1539055780 expire 1539055630 last 1539055553 Oct 08 20:29:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 20:32:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 20:32:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 20:33:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d7d83ab2-527c-7848-9a3b-5ef747e68a27 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809c7569400, cur 1539056020 expire 1539055870 last 1539055793 Oct 08 20:39:05 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 20:39:05 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 08 20:42:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 20:42:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 20:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6ccc2558-5b60-13b4-34e7-f0f76da9dbe4 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d2be1c000, cur 1539056589 expire 1539056439 last 1539056362 Oct 08 20:56:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 20:56:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 20:57:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59bba656-44f8-d54b-ddf8-452afa79d6f0 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881927b9c800, cur 1539057420 expire 1539057270 last 1539057193 Oct 08 20:57:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 20:58:17 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 21:08:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 21:08:43 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 08 21:09:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cfeb1872-c51f-6e3d-1ec0-0c886668d3e7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9e800, cur 1539058196 expire 1539058046 last 1539057969 Oct 08 21:09:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 21:20:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 21:20:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 21:20:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1cf6ef6d-808e-eb6a-9a11-5f0337c64a7e (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b95225c00, cur 1539058851 expire 1539058701 last 1539058624 Oct 08 21:20:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 21:31:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e3edc787-dd98-b595-0c2a-1e0f807ed4c7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e38758800, cur 1539059495 expire 1539059345 last 1539059268 Oct 08 21:31:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 21:35:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 21:35:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 08 21:43:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client feaf4d9d-4859-3d5a-1659-e9f70aeb4956 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe8699000, cur 1539060182 expire 1539060032 last 1539059955 Oct 08 21:43:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 21:46:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca21d097-a095-ce1a-3050-8700484deba8 (at 10.12.4.43@o2ib) Oct 08 21:46:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 21:58:03 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 21:58:03 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 08 22:12:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37589ecf-1ca7-80c6-5d10-330cc3506d83 (at 10.12.4.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880970fbdc00, cur 1539061975 expire 1539061825 last 1539061748 Oct 08 22:12:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 08 22:15:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 09a9e5e5-dc43-8a42-ced0-ab71a4cb5e18 (at 10.12.4.39@o2ib) Oct 08 22:31:00 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 22:31:00 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 08 22:31:45 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 22:44:26 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 22:44:26 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 08 22:49:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7eac84e0-0b8c-56e5-0415-98826936ad68 (at 10.8.28.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880970ffbc00, cur 1539064174 expire 1539064024 last 1539063947 Oct 08 22:53:04 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 22:53:50 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:02:07 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:02:07 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 08 23:10:26 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:10:26 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 08 23:15:00 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:15:00 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 08 23:38:07 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:38:07 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 08 23:38:14 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:38:25 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:38:25 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 08 23:38:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 23:39:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52286e0a-7469-207e-f719-50530b300d60 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807659ed000, cur 1539067168 expire 1539067018 last 1539066941 Oct 08 23:40:10 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:41:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 08 23:42:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e046f394-b90e-fa34-b5bf-34248f6ec092 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880004431800, cur 1539067364 expire 1539067214 last 1539067137 Oct 08 23:45:02 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:45:02 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 08 23:47:05 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:47:05 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 08 23:51:21 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:51:21 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 08 23:55:18 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 08 23:55:18 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 00:01:47 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 00:01:47 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 09 00:12:37 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 00:12:37 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 09 00:27:13 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 00:27:13 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 09 00:40:09 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 00:40:09 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 00:47:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ac13aff-7bfb-d179-9ae2-f72e50f21900 (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881999301400, cur 1539071250 expire 1539071100 last 1539071023 Oct 09 00:48:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 09 00:52:40 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 00:52:40 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 01:05:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f1bd389-24ba-6c3d-ddcf-6f712b99543f (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1f6a9400, cur 1539072348 expire 1539072198 last 1539072121 Oct 09 01:06:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 09 01:10:41 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 01:10:41 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 01:20:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 67b41237-0fc0-458d-7fb9-121dc4fa0afb (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff3d98000, cur 1539073233 expire 1539073083 last 1539073006 Oct 09 01:40:43 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 01:40:43 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 01:43:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 09 01:48:03 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 01:54:09 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 01:54:09 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 09 02:04:34 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 02:43:40 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 02:43:40 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 09 02:48:22 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 02:48:22 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 09 02:53:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 198cef9d-f37d-cfe2-9de3-e92229dbfc00 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7404000, cur 1539078798 expire 1539078648 last 1539078571 Oct 09 02:59:42 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 02:59:42 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 09 03:08:42 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 03:13:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 09 03:19:21 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 03:19:21 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17 previous similar messages Oct 09 03:32:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65e7cb27-728b-65f3-fb9d-cbb1f7cf56b5 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851237400, cur 1539081177 expire 1539081027 last 1539080950 Oct 09 03:43:11 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 03:43:11 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 09 03:49:19 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 03:55:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Oct 09 04:10:01 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 04:10:01 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 09 04:10:56 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 04:15:07 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 04:24:07 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 04:25:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4c8f01f9-be66-e344-0fc4-f4b39942a72c (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800561bb000, cur 1539084339 expire 1539084189 last 1539084112 Oct 09 04:27:09 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 04:27:09 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 09 04:33:51 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 04:46:11 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 04:46:11 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 09 05:00:19 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 05:00:19 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 05:01:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ecb341e4-547a-5ef3-bd15-4e51e2d377a0 (at 10.9.105.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817f072dc00, cur 1539086500 expire 1539086350 last 1539086273 Oct 09 05:24:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f1ee5ce6-514b-2d4c-24b0-0ac8f3361042 (at 10.9.105.4@o2ib4) Oct 09 05:24:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa0220d3-075d-d080-d5c1-c1fb4392f33a (at 10.9.105.2@o2ib4) Oct 09 05:37:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d12d50a-2029-13d3-473f-1bf907d98779 (at 10.9.103.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f85a8f800, cur 1539088676 expire 1539088526 last 1539088449 Oct 09 05:37:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 06:00:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6d12d50a-2029-13d3-473f-1bf907d98779 (at 10.9.103.9@o2ib4) Oct 09 06:06:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 44c8af11-4c8a-f406-4bab-05da4e0803d9 (at 10.12.4.30@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880026c7cc00, cur 1539090399 expire 1539090249 last 1539090172 Oct 09 06:33:08 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 06:33:08 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 06:44:34 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 06:44:34 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 06:45:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 09 06:47:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e3ddb07-e684-4ae8-7d55-31cd0e3d0761 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a98403400, cur 1539092862 expire 1539092712 last 1539092635 Oct 09 06:55:34 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 07:02:35 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 07:02:35 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 07:18:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f391cb18-2dea-a403-02b9-7c06005e60c4 (at 10.9.102.25@o2ib4) Oct 09 07:18:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 781c5682-539a-038e-99a0-d5b3ad05d3fe (at 10.9.102.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880af5e3b000, cur 1539094714 expire 1539094564 last 1539094487 Oct 09 07:34:11 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 07:34:11 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 09 07:47:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43543aab-b1ab-c2ce-f37d-574fd54436ac (at 10.12.4.56@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101bf16c00, cur 1539096475 expire 1539096325 last 1539096248 Oct 09 08:04:59 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 08:12:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 56dc4896-b77d-9a19-8179-017defa860ad (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e8d9b9c00, cur 1539097974 expire 1539097824 last 1539097747 Oct 09 08:35:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 09 08:36:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e19496bd-66b3-e7f4-f90f-f70aa4914ac2 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4c400, cur 1539099394 expire 1539099244 last 1539099167 Oct 09 08:37:51 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 08:40:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 08:42:26 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539099739/real 1539099739] req@ffff88001d62c800 x1610631317123904/t0(0) o104->oak-MDT0000@10.8.0.67@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539099746 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 09 08:42:26 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 09 08:42:34 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539099746/real 1539099746] req@ffff88001d62c800 x1610631317123904/t0(0) o104->oak-MDT0000@10.8.0.67@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539099753 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 08:42:41 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539099754/real 1539099754] req@ffff88001d62c800 x1610631317123904/t0(0) o104->oak-MDT0000@10.8.0.67@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539099761 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 08:42:52 oak-md1-s2 kernel: Lustre: 10492:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539099765/real 1539099765] req@ffff880042016c00 x1610631317176832/t0(0) o104->oak-MDT0000@10.8.0.67@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539099772 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 08:42:52 oak-md1-s2 kernel: Lustre: 10492:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 09 08:43:13 oak-md1-s2 kernel: Lustre: 10492:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539099786/real 1539099786] req@ffff880042016c00 x1610631317176832/t0(0) o104->oak-MDT0000@10.8.0.67@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539099793 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 08:43:13 oak-md1-s2 kernel: Lustre: 10492:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Oct 09 08:43:51 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539099824/real 1539099824] req@ffff88001d62c800 x1610631317123904/t0(0) o104->oak-MDT0000@10.8.0.67@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539099831 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 08:43:51 oak-md1-s2 kernel: Lustre: 10654:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Oct 09 08:44:05 oak-md1-s2 kernel: LustreError: 10654:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.0.67@o2ib6) failed to reply to blocking AST (req@ffff88001d62c800 x1610631317123904 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880f9d2fd000/0x6c00c40209734172 lrc: 4/0,0 mode: PR/PR res: [0x200017cb6:0x11c72:0x0].0x0 bits 0x13 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.0.67@o2ib6 remote: 0x2ee56229a048a37e expref: 2633 pid: 10615 timeout: 7384131263 lvb_type: 0 Oct 09 08:44:05 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.0.67@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 09 08:44:05 oak-md1-s2 kernel: LustreError: 4255:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880038619200 x1610631317487888/t0(0) o104->oak-MDT0000@10.8.0.67@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 09 08:44:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3eac4d27-6938-5bd3-6cf0-77bbdf595f11 (at 10.9.0.61@o2ib4) Oct 09 08:44:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e8f82620-e480-b4c7-f7a4-247bcbefc4a2 (at 10.8.0.68@o2ib6) Oct 09 08:44:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 08:44:40 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 08:45:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ec60b89-87bf-b1a6-cac2-4fb963e823f2 (at 10.9.0.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e20d7b400, cur 1539099926 expire 1539099776 last 1539099699 Oct 09 08:47:32 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 08:48:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f471cc15-fce5-d948-839a-6ac86570f743 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e3c27000, cur 1539100126 expire 1539099976 last 1539099899 Oct 09 08:48:46 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 08:49:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 08:49:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 08:57:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 08:57:21 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 08:57:21 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 08:57:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 389f5328-9875-f202-912f-6953672b8a91 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a887cb800, cur 1539100666 expire 1539100516 last 1539100439 Oct 09 08:58:22 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 08:58:22 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 08:59:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b1609ca4-997c-66b4-96d8-c6982da37213 (at 10.9.106.15@o2ib4) Oct 09 09:01:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b69140d-a83b-c287-c38c-8a22abebae36 (at 10.9.0.62@o2ib4) Oct 09 09:03:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 94770d73-7756-59df-fb6f-e2b5671afc49 (at 10.9.107.4@o2ib4) Oct 09 09:03:22 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 09 09:05:49 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 09:05:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 09:06:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 624a5c85-97b7-3eba-962d-24e77f184fc7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ed2222800, cur 1539101202 expire 1539101052 last 1539100975 Oct 09 09:09:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0e9f7736-8f45-3b9d-bd47-3658160c924a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099ad23800, cur 1539101377 expire 1539101227 last 1539101150 Oct 09 09:13:38 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 09:13:38 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 09:15:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1c349dd2-2d78-e8f7-bd1b-3db7cf812fd0 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e3875fc00, cur 1539101735 expire 1539101585 last 1539101508 Oct 09 09:31:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f3e21df8-9094-3a25-4d26-b928ac0af33e (at 10.12.4.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227cec00, cur 1539102685 expire 1539102535 last 1539102458 Oct 09 09:31:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4b4666da-ba99-1226-e4c3-6c436ffc8157 (at 10.12.4.22@o2ib) Oct 09 09:31:34 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 09 09:32:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43543aab-b1ab-c2ce-f37d-574fd54436ac (at 10.12.4.56@o2ib) Oct 09 09:32:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 09:37:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 962d37a0-2778-93b7-9dd8-8e6ceef4b192 (at 10.8.24.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe97c6400, cur 1539103026 expire 1539102876 last 1539102799 Oct 09 09:41:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 13b286f4-17a5-9ffd-ddd2-f347666309f9 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a0400, cur 1539103297 expire 1539103147 last 1539103070 Oct 09 09:42:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 09 09:42:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 09:44:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 177cd922-7c60-1360-3934-76d615c1c9f8 (at 10.8.20.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4bc00, cur 1539103460 expire 1539103310 last 1539103233 Oct 09 09:46:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 09 09:48:38 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 09:49:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90c73117-5a1a-a601-2d02-08380b787870 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804e1749000, cur 1539103746 expire 1539103596 last 1539103519 Oct 09 09:49:06 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 09 09:58:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1edce57f-2b73-efe7-e2dc-6baf74838141 (at 10.8.24.13@o2ib6) Oct 09 10:09:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59dea6b4-2bbf-9cad-6f7b-002e3e08717e (at 10.8.30.22@o2ib6) Oct 09 10:09:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 10:14:42 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 10:16:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 092ee32d-2360-2274-589d-667ced5b055e (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881849fd7c00, cur 1539105402 expire 1539105252 last 1539105175 Oct 09 10:17:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1d908f41-b809-cd21-bf22-7ee32c8bb207 (at 10.8.15.10@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff880d28643000, cur 1539105478 expire 1539105328 last 1539105256 Oct 09 10:17:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 10:21:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c27b409e-7ed5-a2f3-a748-4d07b47ab351 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab4308000, cur 1539105675 expire 1539105525 last 1539105448 Oct 09 10:27:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d162cfd9-5986-43fc-9564-2f4c0f291a56 (at 10.8.4.4@o2ib6) Oct 09 10:27:14 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Oct 09 10:38:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Oct 09 10:38:41 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 10:46:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e71b407e-b6eb-28ef-7147-41e9f85c2c1f (at 10.8.20.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881816ee1000, cur 1539107177 expire 1539107027 last 1539106950 Oct 09 10:52:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Oct 09 11:05:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a2574b5-ce5d-b821-8651-1feeb847c072 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef2878400, cur 1539108304 expire 1539108154 last 1539108077 Oct 09 11:11:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 09 11:11:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f77fd143-2b5d-a915-5503-29e45f2a4c71 (at 10.0.2.3@o2ib5) reconnecting Oct 09 11:24:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 17f3802a-acb7-695a-5617-9e93660e1036 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb981f400, cur 1539109487 expire 1539109337 last 1539109260 Oct 09 11:27:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 09 11:27:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 11:36:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a60c311-e84d-cf92-45f6-6a653054626e (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818292ad800, cur 1539110173 expire 1539110023 last 1539109946 Oct 09 11:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bccf629c-900e-c4a9-171a-d70129138265 (at 10.8.14.1@o2ib6) Oct 09 11:38:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 11:51:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fbdf66ba-c855-ca42-c36a-de9608b6bc09 (at 10.8.22.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88048a9e8c00, cur 1539111086 expire 1539110936 last 1539110859 Oct 09 11:52:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 07e82949-09f2-77c6-282d-eab2c63b04c7 (at 10.8.10.3@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef5000, cur 1539111162 expire 1539111012 last 1539110936 Oct 09 11:52:42 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Oct 09 11:53:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 57293d75-146b-78c3-a1e3-e9ec35f151d6 (at 10.8.22.9@o2ib6) in 216 seconds. I think it's dead, and I am evicting it. exp ffff880ee604b400, cur 1539111238 expire 1539111088 last 1539111022 Oct 09 11:53:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 11:54:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 09 12:02:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5192176e-8874-a996-5e67-180085d3bde4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd7a7c000, cur 1539111759 expire 1539111609 last 1539111532 Oct 09 12:02:39 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 12:05:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43c12f45-690c-c86a-5393-17436942b00b (at 10.8.6.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28d800, cur 1539111918 expire 1539111768 last 1539111691 Oct 09 12:05:18 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 12:06:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbb9db85-60f5-adbd-5b0f-c1781850eda7 (at 10.8.19.8@o2ib6) Oct 09 12:06:39 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 12:16:10 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 12:23:32 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 12:26:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2813e1af-2783-9eb8-c2ec-a044fc7c8bb7 (at 10.9.103.17@o2ib4) Oct 09 12:26:35 oak-md1-s2 kernel: Lustre: Skipped 13 previous similar messages Oct 09 12:27:17 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 12:31:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 759d33c9-ef2c-0b1c-a65c-14fe67b83348 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacc5fc00, cur 1539113516 expire 1539113366 last 1539113289 Oct 09 12:51:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 09 12:51:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 12:52:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 338c1b50-57be-58b1-f9d9-ca00d22d3a29 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca3800, cur 1539114732 expire 1539114582 last 1539114505 Oct 09 12:52:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 12:56:28 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 12:56:28 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 12:57:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 09 12:58:00 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 12:58:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 379331aa-319a-fdd9-f216-d677715fcccf (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88181977a000, cur 1539115118 expire 1539114968 last 1539114891 Oct 09 12:59:17 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 13:07:05 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 13:07:05 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 13:09:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce5d1c13-0049-c5ae-7e14-505e8c87ee3d (at 10.8.20.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005dcb0800, cur 1539115775 expire 1539115625 last 1539115548 Oct 09 13:10:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 09 13:10:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cacffd13-3481-1798-7ae6-026667d94c82 (at 10.8.17.26@o2ib6) in 204 seconds. I think it's dead, and I am evicting it. exp ffff881927b9bc00, cur 1539115851 expire 1539115701 last 1539115647 Oct 09 13:22:49 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 13:23:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Oct 09 13:23:57 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 13:23:57 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 13:43:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c55ca29d-9809-7f68-a8fd-9ca5efcd56e0 (at 10.8.20.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa3f2000, cur 1539117791 expire 1539117641 last 1539117564 Oct 09 13:43:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 13:47:23 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 13:47:23 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 13:50:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c07c9617-ba47-0326-c28a-52943fcae932 (at 10.8.20.4@o2ib6) Oct 09 13:59:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 14:00:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4d343412-7c87-63b1-653b-5ed6020185ff (at 10.8.17.12@o2ib6) Oct 09 14:00:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2f90e04-eb62-5f29-a63c-88080c7107b0 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039232800, cur 1539118847 expire 1539118697 last 1539118620 Oct 09 14:01:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Oct 09 14:03:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 14:04:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 27ea8ca0-cdd0-12f8-ed1a-4f222bb356f8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818feafc800, cur 1539119052 expire 1539118902 last 1539118825 Oct 09 14:06:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 14:07:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 328bfdcd-c758-f907-9339-1a3dfcb3f2e6 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc0809c00, cur 1539119272 expire 1539119122 last 1539119045 Oct 09 14:09:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 14:11:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b988b7e-92a5-e3c5-7611-ab2f9b0ed175 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881beeebe000, cur 1539119461 expire 1539119311 last 1539119234 Oct 09 14:13:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 14:14:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 970b81d9-8c0d-0189-7259-8c5cc0a5c905 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200befc800, cur 1539119643 expire 1539119493 last 1539119416 Oct 09 14:16:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 09 14:17:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f6bedfdf-d232-d24a-5648-d2295e627285 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf1ea0c00, cur 1539119845 expire 1539119695 last 1539119618 Oct 09 14:18:03 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 14:18:03 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 09 14:21:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a8cc75b-5f80-36a0-bba1-684353db8784 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881765a82000, cur 1539120067 expire 1539119917 last 1539119840 Oct 09 14:31:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Oct 09 14:38:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43edb03d-f62c-e463-4f94-20fc02fc5ffe (at 10.8.21.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afa000, cur 1539121102 expire 1539120952 last 1539120875 Oct 09 14:45:02 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 14:45:02 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 14:51:07 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 14:51:07 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 14:59:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43edb03d-f62c-e463-4f94-20fc02fc5ffe (at 10.8.21.35@o2ib6) Oct 09 15:00:39 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:03:57 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:03:57 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 15:13:43 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:13:43 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 15:14:51 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:18:25 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:20:19 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:20:19 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 09 15:22:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6fe5b944-ed01-0a14-7f9f-27239ab218c8 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184f70ac00, cur 1539123746 expire 1539123596 last 1539123519 Oct 09 15:28:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7167cdc5-4f77-1cd6-981d-9be6bc64759a (at 10.9.104.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c5bb8f800, cur 1539124125 expire 1539123975 last 1539123898 Oct 09 15:38:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 09 15:38:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bfce3c3-9f01-dade-869d-6d1ff1be1516 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daca0b400, cur 1539124691 expire 1539124541 last 1539124464 Oct 09 15:38:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 15:39:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f8df5e4-c1a9-714d-97c9-ccae11dc440b (at 10.8.27.13@o2ib6) in 186 seconds. I think it's dead, and I am evicting it. exp ffff881eccec9c00, cur 1539124767 expire 1539124617 last 1539124581 Oct 09 15:45:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e58be353-6bc0-1c69-b54d-1db7c01f7d82 (at 10.9.112.17@o2ib4) Oct 09 15:50:06 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:50:06 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 09 15:50:19 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:50:19 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 15:51:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c40acc56-e601-fdea-a296-bcbeada9b358 (at 10.9.104.42@o2ib4) Oct 09 15:52:20 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 15:52:20 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 16:00:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f40e0974-1613-89e7-33df-71cafaaff4af (at 10.8.27.20@o2ib6) Oct 09 16:00:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 16:01:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f9f8903b-0cd9-810a-179c-528ea7dfa54b (at 10.8.27.13@o2ib6) Oct 09 16:01:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 83e1b268-93b0-cdeb-4bb7-f64fec1e15c7 (at 10.8.27.16@o2ib6) Oct 09 16:06:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 09 16:06:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 16:08:07 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:08:07 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 16:12:03 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:12:03 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 16:14:15 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:14:15 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 16:21:22 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:21:22 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 09 16:25:16 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:25:16 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 16:26:42 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:26:42 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 09 16:30:15 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:35:04 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:35:04 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 16:40:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e8db1c1-19c3-bae2-ace7-f26790b52295 (at 10.8.25.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e19c40c00, cur 1539128442 expire 1539128292 last 1539128215 Oct 09 16:40:42 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 09 16:45:49 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:45:49 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 09 16:55:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b3723f1-0eea-1ff9-df62-0a27195fa17b (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d2692c00, cur 1539129330 expire 1539129180 last 1539129103 Oct 09 16:56:36 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 16:56:36 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 09 17:06:37 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 17:06:37 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 09 17:17:26 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 17:17:26 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 09 17:18:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:22:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e178c39f-47cc-18c5-0768-4723df3c24ef (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ee801000, cur 1539130956 expire 1539130806 last 1539130729 Oct 09 17:22:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb16a738-c5e8-0837-ac83-a869ecfc05ce (at 10.8.25.23@o2ib6) Oct 09 17:26:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ed9f4e5c-3394-0abf-e95f-8ab06aa6ca0e (at 10.12.4.26@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d7800, cur 1539131201 expire 1539131051 last 1539130974 Oct 09 17:27:46 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 17:27:46 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 09 17:34:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:37:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:38:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2a26d9d-855d-137c-827a-e132da521175 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812032e8400, cur 1539131922 expire 1539131772 last 1539131695 Oct 09 17:40:26 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 17:40:26 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 09 17:40:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:41:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4c6311ac-4a9e-490e-ced7-2d3c7b9d650c (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881849fd7c00, cur 1539132093 expire 1539131943 last 1539131866 Oct 09 17:47:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 809d2034-227b-2703-8474-4d3af37846ae (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff0b33400, cur 1539132470 expire 1539132320 last 1539132243 Oct 09 17:48:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:51:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:51:55 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 17:51:55 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 09 17:52:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d60edd9-da9c-d285-12d2-891e30650ea5 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e902bb400, cur 1539132737 expire 1539132587 last 1539132510 Oct 09 17:54:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:55:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 219528ab-e619-a766-0846-ab9568610bea (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185bacb400, cur 1539132919 expire 1539132769 last 1539132692 Oct 09 17:57:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 17:58:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa9b5b5e-365b-bbf7-75ae-3d718ff2203a (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b2b4c800, cur 1539133114 expire 1539132964 last 1539132887 Oct 09 18:02:27 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 18:02:27 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 09 18:04:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 470bfe3a-13c3-86d9-0128-4c250723ccf1 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0aa400, cur 1539133474 expire 1539133324 last 1539133247 Oct 09 18:04:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 18:05:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d69e557-799e-044a-652c-35b576d2bb7c (at 10.8.1.4@o2ib6) in 165 seconds. I think it's dead, and I am evicting it. exp ffff88185baef400, cur 1539133550 expire 1539133400 last 1539133385 Oct 09 18:12:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d52e9a7f-3a4c-5479-e46e-c67d258b5ede (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e2c00, cur 1539133971 expire 1539133821 last 1539133744 Oct 09 18:13:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 18:17:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 09 18:18:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 218dcf38-ef19-ce71-9c07-75b1a8a22b5d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee1a00800, cur 1539134295 expire 1539134145 last 1539134068 Oct 09 18:24:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ed7661d-de8e-6a17-a9e2-e7d56e30d0e1 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804e1748c00, cur 1539134661 expire 1539134511 last 1539134434 Oct 09 18:24:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 18:28:15 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 18:28:15 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 09 18:28:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 09 18:28:58 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 09 18:30:03 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 18:30:03 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 18:33:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 88f8263b-fd2c-1ded-1c87-5eabaec7cf4e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daab0b400, cur 1539135199 expire 1539135049 last 1539134972 Oct 09 18:33:19 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 09 18:39:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 09 18:39:14 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 09 18:48:22 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 18:48:22 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 18:49:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a1c4b5b-f0ec-5d12-0d9c-911ac1bbec0d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7597c00, cur 1539136148 expire 1539135998 last 1539135921 Oct 09 18:49:08 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 09 18:49:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 09 18:49:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 19:04:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f859ad8-af97-cc56-8fc2-73548a2ddac2 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e61522800, cur 1539137064 expire 1539136914 last 1539136837 Oct 09 19:04:24 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 09 19:10:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 19:10:44 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 09 19:23:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9d67afd0-987f-a972-86df-62e2ec080222 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9a400, cur 1539138197 expire 1539138047 last 1539137970 Oct 09 19:26:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 19:32:34 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 19:32:34 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 19:32:36 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 19:32:36 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 09 19:33:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd9083b1-5e69-1091-48d5-75c921b7295b (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb82c00, cur 1539138815 expire 1539138665 last 1539138588 Oct 09 19:34:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 19:38:57 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 19:38:57 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 09 19:40:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 19:41:09 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 19:41:09 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 19:47:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da3f041d-549d-1611-2fd3-47e237ca3105 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019ce1e400, cur 1539139643 expire 1539139493 last 1539139416 Oct 09 19:47:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 19:48:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 19:57:01 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 19:59:05 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539140338/real 1539140338] req@ffff881a35799800 x1610631486619456/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539140345 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 09 19:59:05 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Oct 09 19:59:20 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539140352/real 1539140352] req@ffff881a35799800 x1610631486619456/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539140359 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 19:59:20 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 09 19:59:41 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539140374/real 1539140374] req@ffff881a35799800 x1610631486619456/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539140381 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 19:59:41 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 09 20:00:23 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539140416/real 1539140416] req@ffff881a35799800 x1610631486619456/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539140423 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 20:00:23 oak-md1-s2 kernel: Lustre: 10614:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 09 20:01:27 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 20:01:33 oak-md1-s2 kernel: LustreError: 10614:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.17.27@o2ib6) failed to reply to blocking AST (req@ffff881a35799800 x1610631486619456 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880e37790400/0x6c00c4048de14f30 lrc: 4/0,0 mode: PR/PR res: [0x200017e8f:0x10:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.8.17.27@o2ib6 remote: 0xd9de9011cdcb0d9d expref: 3006 pid: 10556 timeout: 7424829312 lvb_type: 0 Oct 09 20:01:33 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.17.27@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 09 20:01:41 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 20:01:41 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 20:02:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:06:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:06:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7da89dab-8f9a-2c4e-a9c9-4bac16e0bfd5 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ca4f2c800, cur 1539140795 expire 1539140645 last 1539140568 Oct 09 20:06:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 20:08:01 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 20:08:01 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 20:11:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fd8236e-050b-c62d-2664-f6711708a9ca (at 10.8.29.6@o2ib6) Oct 09 20:11:53 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 20:11:53 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 20:14:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:20:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:20:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97fb956c-3b82-65cc-7484-424a5a2e29d9 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8813511ab800, cur 1539141644 expire 1539141494 last 1539141417 Oct 09 20:20:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 20:23:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:30:14 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539142206/real 1539142206] req@ffff8815d0f16600 x1610631492809600/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539142213 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 09 20:30:14 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 09 20:30:28 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539142221/real 1539142221] req@ffff8815d0f16600 x1610631492809600/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539142228 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 20:30:28 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 09 20:30:49 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539142242/real 1539142242] req@ffff8815d0f16600 x1610631492809600/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539142249 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 20:30:49 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 09 20:31:31 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539142284/real 1539142284] req@ffff8815d0f16600 x1610631492809600/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539142291 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 09 20:31:31 oak-md1-s2 kernel: Lustre: 6662:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 09 20:32:41 oak-md1-s2 kernel: LustreError: 6662:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.17.27@o2ib6) failed to reply to blocking AST (req@ffff8815d0f16600 x1610631492809600 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8807baab8600/0x6c00c404b8247c2a lrc: 4/0,0 mode: PR/PR res: [0x20000dae5:0x1554a:0x0].0x0 bits 0x13 rrc: 72 type: IBT flags: 0x60200400000020 nid: 10.8.17.27@o2ib6 remote: 0x89ddea59882d39c3 expref: 3006 pid: 10658 timeout: 7426697489 lvb_type: 0 Oct 09 20:32:41 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.17.27@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 09 20:32:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:32:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 20:36:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea515c62-9266-6629-1362-1a61c98db4bf (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d39b4ec00, cur 1539142598 expire 1539142448 last 1539142371 Oct 09 20:36:38 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 20:43:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:43:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 20:46:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48c2b764-d8f0-0eb0-ac38-82dd59bdf99d (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812642ac000, cur 1539143217 expire 1539143067 last 1539142990 Oct 09 20:46:57 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 20:58:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aa0c17db-e198-c07a-133f-792a52b7f97d (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4284d000, cur 1539143907 expire 1539143757 last 1539143680 Oct 09 20:59:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 20:59:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 21:09:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e596644-5e34-721b-8ff3-46106e8c7ee8 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef287fc00, cur 1539144555 expire 1539144405 last 1539144328 Oct 09 21:09:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 21:11:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 21:11:06 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 09 21:24:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b290fe1c-dc85-cb2e-66c8-4411b7bed07e (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809a6fed400, cur 1539145469 expire 1539145319 last 1539145242 Oct 09 21:27:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 21:34:53 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 21:34:53 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 21:36:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99244144-d6a4-c481-cd78-3189b73c544c (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f593dac00, cur 1539146180 expire 1539146030 last 1539145953 Oct 09 21:52:46 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 21:52:46 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 21:56:53 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 21:56:53 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 09 22:11:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6f22f7a-4598-5666-326f-e5d01b42f3fc (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa873800, cur 1539148305 expire 1539148155 last 1539148078 Oct 09 22:14:08 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:14:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b0f84b1-b636-9cd6-636b-47f8fd27a775 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186c672400, cur 1539148496 expire 1539148346 last 1539148269 Oct 09 22:16:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 22:16:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 22:17:14 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:19:12 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:19:12 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 09 22:21:47 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:25:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 14d57d78-7020-346b-0130-dfa780d60d30 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88196caf0c00, cur 1539149109 expire 1539148959 last 1539148882 Oct 09 22:25:15 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:25:15 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 09 22:25:59 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:29:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 22:31:59 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:31:59 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 09 22:36:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 040f67a6-c0ef-8205-9d85-e608f1e6e062 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015ca5c00, cur 1539149796 expire 1539149646 last 1539149569 Oct 09 22:36:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 22:39:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 09 22:44:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 09 22:44:58 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:48:03 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:55:30 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:57:17 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 22:57:17 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 23:03:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9df46d20-859b-e8ca-5d9d-9eb310d1fbce (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88196caf0000, cur 1539151420 expire 1539151270 last 1539151193 Oct 09 23:03:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 23:06:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:08:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9244b6ed-9e87-627d-5e92-85551d6c1a34 (at 10.8.13.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819da3f1400, cur 1539151684 expire 1539151534 last 1539151457 Oct 09 23:09:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:09:56 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:10:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f166efd5-500d-242b-16b7-0f1bedb38438 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b13238c00, cur 1539151842 expire 1539151692 last 1539151615 Oct 09 23:13:10 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:14:09 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:14:09 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 23:18:07 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:19:23 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:20:39 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:20:39 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 09 23:22:50 oak-md1-s2 kernel: Lustre: 10508:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:22:50 oak-md1-s2 kernel: Lustre: 10508:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 23:26:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79f60cad-0ba4-181b-8f03-b5e6b2788ce6 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe937d400, cur 1539152815 expire 1539152665 last 1539152588 Oct 09 23:27:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:27:46 oak-md1-s2 kernel: Lustre: 10566:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:27:46 oak-md1-s2 kernel: Lustre: 10566:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 09 23:28:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 09 23:28:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f47eacfe-91c7-5c93-c30e-128d11d8d114 (at 10.8.17.28@o2ib6) in 219 seconds. I think it's dead, and I am evicting it. exp ffff880b28fdbc00, cur 1539152891 expire 1539152741 last 1539152672 Oct 09 23:30:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 44875561-38e5-8c60-fee4-75792ae2c760 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802da794400, cur 1539153049 expire 1539152899 last 1539152822 Oct 09 23:33:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 93449d4b-2a14-baac-8aeb-2e675cde5854 (at 10.8.13.3@o2ib6) Oct 09 23:33:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:36:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:37:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cd41b237-8e07-c59b-ac4f-104719153aaf (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7403400, cur 1539153455 expire 1539153305 last 1539153228 Oct 09 23:39:24 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 09 23:39:24 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 09 23:39:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:42:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:43:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f3dbb7c2-ab0f-6fb8-08e6-712d97f725a5 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff09bc400, cur 1539153807 expire 1539153657 last 1539153580 Oct 09 23:43:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 09 23:45:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:52:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 09 23:56:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 83bd17a6-f5a2-a218-c5eb-1aaa0479a9cf (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca4000, cur 1539154567 expire 1539154417 last 1539154340 Oct 09 23:56:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 00:02:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 00:02:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 00:12:48 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 00:12:48 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 00:31:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 491102b7-2d8a-7bb7-8ae2-658fd0be563b (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc6a400, cur 1539156697 expire 1539156547 last 1539156470 Oct 10 00:31:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 00:32:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 00:36:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 00:36:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 378671c2-fbd7-e0ce-5d55-4d7a92df8f4b (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a2f5000, cur 1539157004 expire 1539156854 last 1539156777 Oct 10 00:40:44 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 00:40:44 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 00:47:26 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 00:47:26 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 00:47:45 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 00:47:45 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 00:50:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6bdd5a9d-8bdd-4ff7-d5c2-79876ed98971 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817f072e800, cur 1539157820 expire 1539157670 last 1539157593 Oct 10 00:51:14 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 00:51:14 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 00:51:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6e7dbe8-678f-5bec-58d6-4a582a6d69c2 (at 10.8.4.29@o2ib6) Oct 10 00:55:20 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 00:55:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 22ef95d4-dc78-075b-c533-3cb95b502559 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3bcc00, cur 1539158151 expire 1539158001 last 1539157924 Oct 10 00:55:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 00:58:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 00:58:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 00:58:39 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 00:58:39 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 01:03:44 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 01:03:44 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 01:14:58 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 01:14:58 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 10 01:16:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e3ba65df-8811-2767-94e3-b093b1e22ff0 (at 10.9.103.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e7ba800, cur 1539159386 expire 1539159236 last 1539159159 Oct 10 01:28:41 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 01:28:41 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 01:32:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6b153d4-1acf-40a4-f40c-dae5111ee4e2 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd869fc00, cur 1539160342 expire 1539160192 last 1539160115 Oct 10 01:33:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 01:41:08 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 01:41:08 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 01:41:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e3ba65df-8811-2767-94e3-b093b1e22ff0 (at 10.9.103.2@o2ib4) Oct 10 01:43:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cdcba549-9448-0951-46e6-497f23805293 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88152c641000, cur 1539161036 expire 1539160886 last 1539160809 Oct 10 01:43:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 01:47:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 01:53:28 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 01:53:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 284afda2-450f-fd57-ea68-b853668aba28 (at 10.8.3.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dab766800, cur 1539161632 expire 1539161482 last 1539161405 Oct 10 01:54:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ebcc7456-5e8f-e275-1e09-9882443932f1 (at 10.8.3.32@o2ib6) Oct 10 01:57:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5025dce0-023b-4b33-0eef-9f835ed1fe42 (at 10.9.103.10@o2ib4) Oct 10 02:00:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb73bdd4-f780-5026-d034-e0f8f9406d0c (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005b9a7800, cur 1539162030 expire 1539161880 last 1539161803 Oct 10 02:00:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 02:04:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 02:04:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 62182baa-051c-f41d-d7eb-ce02a94e3712 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb9818c00, cur 1539162296 expire 1539162146 last 1539162069 Oct 10 02:09:32 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 02:09:32 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 02:12:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 19448893-dd66-3145-5c83-5b17e66d82b9 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9440400, cur 1539162774 expire 1539162624 last 1539162547 Oct 10 02:15:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 02:18:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c69c7ec1-ad3b-385a-a047-f6d0ebd091fe (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881927b9b800, cur 1539163136 expire 1539162986 last 1539162909 Oct 10 02:20:50 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 02:20:50 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Oct 10 02:24:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99195567-f8f7-248b-8910-5bf93daafe19 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816d4e06800, cur 1539163489 expire 1539163339 last 1539163262 Oct 10 02:25:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 02:25:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 02:28:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94c8cbd4-8bed-167f-e57b-6e3853827ce8 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881486024400, cur 1539163737 expire 1539163587 last 1539163510 Oct 10 02:32:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f6db5940-0b77-214a-ed86-0a5bebf64cbf (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac1a3400, cur 1539163926 expire 1539163776 last 1539163699 Oct 10 02:32:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 02:37:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 10 02:37:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 02:37:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d2a295aa-5872-24a4-b2e1-8f13b512348b (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f196e5400, cur 1539164222 expire 1539164072 last 1539163995 Oct 10 02:43:10 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 02:43:10 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 10 02:43:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2a65bbc-6679-b0f6-63fb-323f73061e2a (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa844c00, cur 1539164598 expire 1539164448 last 1539164371 Oct 10 02:46:23 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 02:46:23 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 02:49:56 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 02:49:56 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 02:51:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6fa082ba-9b5c-09f4-0ea4-8421d7169bc1 (at 10.9.103.1@o2ib4) Oct 10 02:51:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 02:55:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9efe21e0-ec3c-9dd3-1f1d-ac80feacafab (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa876c00, cur 1539165316 expire 1539165166 last 1539165089 Oct 10 02:59:14 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 02:59:14 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 03:03:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 98697bfc-793f-4b11-46e7-9a9d15be7a2c (at 10.8.6.19@o2ib6) Oct 10 03:03:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 03:09:47 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 03:09:47 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 03:31:43 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 03:31:43 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 03:33:16 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 03:33:16 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 10 03:36:59 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 03:36:59 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 10 04:00:55 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:00:55 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 04:02:27 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:02:27 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 04:06:40 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:19:58 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:19:58 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 04:21:03 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:28:01 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:28:01 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 04:32:42 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:41:24 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:50:51 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 04:50:51 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 05:03:07 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 05:06:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 35db2d6b-44fe-b90e-6111-701958ebdb63 (at 10.8.24.24@o2ib6) Oct 10 05:07:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f60a7882-8b05-af09-16df-0b088d5ded51 (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582301400, cur 1539173220 expire 1539173070 last 1539172993 Oct 10 05:07:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 05:16:37 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 05:16:37 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 05:36:51 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 05:36:51 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 05:47:41 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 05:47:41 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 05:53:21 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 05:53:21 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 05:59:02 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 05:59:02 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 10 06:10:20 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 06:10:20 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 10 06:30:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 10 06:31:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df895741-c76a-92ab-aa24-38b509477b9c (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004a19f000, cur 1539178277 expire 1539178127 last 1539178050 Oct 10 06:42:47 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 06:42:47 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 06:47:20 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 06:47:20 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 06:51:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d1e05d8b-9c73-f7a5-7f27-95107aaae5b8 (at 10.8.3.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88148eef0400, cur 1539179483 expire 1539179333 last 1539179256 Oct 10 06:52:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a4a3f13b-121a-9bfc-dd6e-277ee301c7e0 (at 10.8.3.4@o2ib6) Oct 10 07:09:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 227f3283-f7c1-6f23-0ac5-9ada97a9a7b3 (at 10.12.4.30@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018dd75400, cur 1539180559 expire 1539180409 last 1539180332 Oct 10 07:10:52 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:12:41 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:23:48 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:24:31 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:24:31 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 07:33:42 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:33:42 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 07:35:39 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:35:39 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 07:36:17 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:36:17 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 07:37:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a2ad189e-6fe1-9844-413f-38174bd0976d (at 10.8.8.35@o2ib6) Oct 10 07:38:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9dc94cf4-39af-b4a0-d487-11da89ced44e (at 10.8.8.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880067358000, cur 1539182326 expire 1539182176 last 1539182099 Oct 10 07:42:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa56e78e-edde-dec0-ea12-ef897c509d0f (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817e0f15000, cur 1539182538 expire 1539182388 last 1539182311 Oct 10 07:46:46 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 07:46:46 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 08:09:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 10 08:15:02 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 08:15:02 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 08:15:39 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 08:19:16 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 08:19:16 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 08:22:56 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 08:22:56 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 08:27:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b76cb23-14f4-9dca-f691-df69b0bc401f (at 10.9.106.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227ca400, cur 1539185232 expire 1539185082 last 1539185005 Oct 10 08:29:14 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 08:39:14 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 08:39:14 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 10 08:40:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 35db2d6b-44fe-b90e-6111-701958ebdb63 (at 10.8.24.24@o2ib6) Oct 10 08:41:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0439e0bc-f3ce-94c4-de8b-fc76309f7d50 (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818987dec00, cur 1539186089 expire 1539185939 last 1539185862 Oct 10 08:50:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48eb8828-9a65-3d65-3181-aff21a995713 (at 10.8.18.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c649400, cur 1539186644 expire 1539186494 last 1539186417 Oct 10 08:52:10 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 08:52:10 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 08:52:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4628f64b-3b31-c146-eaad-483ff35ac07c (at 10.9.106.7@o2ib4) Oct 10 09:12:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to da81cd8a-54bb-1bcc-ecce-162331019b10 (at 10.9.103.5@o2ib4) Oct 10 09:16:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 35db2d6b-44fe-b90e-6111-701958ebdb63 (at 10.8.24.24@o2ib6) Oct 10 09:16:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05dcee20-aba9-89d6-bcc6-67058c43a5ce (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4b597c00, cur 1539188188 expire 1539188038 last 1539187961 Oct 10 09:16:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d28ac457-8b45-1c3b-209e-882b65bf1f88 (at 10.8.18.24@o2ib6) Oct 10 09:20:03 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 09:20:03 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 10 09:30:02 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 09:30:02 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 10 09:51:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a46da61-f39a-afa0-af5d-f04bbc02fb9d (at 10.8.4.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe774a000, cur 1539190307 expire 1539190157 last 1539190080 Oct 10 09:58:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) Oct 10 09:59:43 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 10:01:29 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 10:01:29 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 10:02:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 10 10:04:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e6bd271e-bb17-0fb2-9a1c-e73181ece7fe (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d6000, cur 1539191058 expire 1539190908 last 1539190831 Oct 10 10:21:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Oct 10 10:21:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Oct 10 10:22:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fd8236e-050b-c62d-2664-f6711708a9ca (at 10.8.29.6@o2ib6) Oct 10 10:22:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 10 10:26:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c994c48d-a573-4705-bebc-097177fe8687 (at 10.8.29.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063423c00, cur 1539192401 expire 1539192251 last 1539192174 Oct 10 10:26:41 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 10:27:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) Oct 10 10:27:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 10:28:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 73bd384a-ce4f-ed45-ddd4-df4c88792f0e (at 10.8.10.8@o2ib6) Oct 10 10:29:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Oct 10 10:29:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 10:39:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 10 10:39:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 10:40:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ca41ff1-4cfc-80ed-b80d-8259124e4433 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf017c00, cur 1539193211 expire 1539193061 last 1539192984 Oct 10 10:43:37 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 10:43:37 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 10:49:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 10 10:50:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d4845f7-235e-d7e4-6a94-6bec005f8145 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200be4d400, cur 1539193804 expire 1539193654 last 1539193577 Oct 10 10:53:39 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 10:59:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 10 11:00:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a1510e0-214c-25da-33e1-d44978fd87b1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b35310000, cur 1539194421 expire 1539194271 last 1539194194 Oct 10 11:10:30 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 11:10:30 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 11:12:13 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 11:22:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53446d39-6437-7492-c204-071790279ef0 (at 10.8.18.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa012c00, cur 1539195776 expire 1539195626 last 1539195549 Oct 10 11:28:22 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 11:48:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a0ee5a30-92fd-dfba-d31f-5c82ad81f21b (at 10.8.4.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef102c00, cur 1539197325 expire 1539197175 last 1539197098 Oct 10 11:49:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dafb6a59-e11b-03ee-af38-b4b95700f3c7 (at 10.8.4.22@o2ib6) Oct 10 11:53:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99ae5692-4652-15d1-5a33-3d0ffb9589be (at 10.8.18.11@o2ib6) Oct 10 12:01:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ddfa55e-7840-40f5-d97e-115b14bdbcf6 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014e25bc00, cur 1539198101 expire 1539197951 last 1539197874 Oct 10 12:01:55 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:01:55 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 12:03:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 10 12:04:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b962a244-dac1-e0a2-6726-cb866f2f7b52 (at 10.9.103.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6c8d8800, cur 1539198259 expire 1539198109 last 1539198032 Oct 10 12:07:57 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:08:35 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:08:35 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 12:14:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3fd8236e-050b-c62d-2664-f6711708a9ca (at 10.8.29.6@o2ib6) Oct 10 12:15:50 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:15:50 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 12:17:35 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:19:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b218481-f844-0bcb-4314-9f314ef72e3a (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff09be000, cur 1539199140 expire 1539198990 last 1539198913 Oct 10 12:19:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 10 12:21:02 oak-md1-s2 kernel: Lustre: 10508:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:21:48 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:23:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0bf74055-d5fb-088f-d369-8978f44cc8bb (at 10.8.28.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28a000, cur 1539199406 expire 1539199256 last 1539199179 Oct 10 12:25:56 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:28:37 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:28:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b962a244-dac1-e0a2-6726-cb866f2f7b52 (at 10.9.103.16@o2ib4) Oct 10 12:32:53 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:39:22 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:39:22 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 10 12:44:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7748ded-adcc-8618-45ae-661e58335242 (at 10.8.28.4@o2ib6) Oct 10 12:45:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 317e5427-a261-27ae-c714-a0c4acf9c974 (at 10.9.106.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bee61d800, cur 1539200712 expire 1539200562 last 1539200485 Oct 10 12:50:53 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 12:50:53 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 10 13:07:25 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 13:07:25 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Oct 10 13:11:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 342b0562-28c3-f7d6-a65a-02b91d70985c (at 10.9.106.6@o2ib4) Oct 10 13:17:33 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 13:17:33 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 10 13:31:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b908bcea-ba07-d75b-9845-c372522807e6 (at 10.9.106.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000c64a400, cur 1539203513 expire 1539203363 last 1539203286 Oct 10 13:31:55 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 13:31:55 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 10 13:45:11 oak-md1-s2 kernel: LNet: 206745:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.224@o2ib5 version 12/12 incarnation 1538318958121204/1539204309797048 Oct 10 13:45:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42ddaac4-4249-5052-00c0-3d0a27940cdb (at 10.0.2.224@o2ib5) Oct 10 13:47:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e68002cf-86cc-964d-74ea-0c6854b19bfe (at 10.0.2.224@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881596264400, cur 1539204449 expire 1539204299 last 1539204222 Oct 10 13:53:59 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 13:53:59 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 10 13:55:59 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 13:57:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4da800c5-ba23-6cfa-3a59-f26dd81aa7e0 (at 10.9.106.11@o2ib4) Oct 10 14:00:03 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 14:00:03 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 14:06:12 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 14:06:12 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 10 14:16:39 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 14:16:39 oak-md1-s2 kernel: Lustre: 10552:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 10 14:27:12 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 14:27:12 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 23 previous similar messages Oct 10 14:40:47 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 14:40:47 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 23 previous similar messages Oct 10 14:42:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 435976f3-3f61-f2d5-0cc2-f54a68c7fee1 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005771cc00, cur 1539207731 expire 1539207581 last 1539207504 Oct 10 14:44:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa0a1440-c34e-7f04-afaf-afb7e9606852 (at 10.9.107.36@o2ib4) Oct 10 14:46:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cf713059-26fd-80d5-e6c6-f4adf97a36ea (at 10.9.107.36@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200bdf6c00, cur 1539207960 expire 1539207810 last 1539207733 Oct 10 14:53:05 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 14:53:05 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 10 14:53:21 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539208394/real 1539208394] req@ffff880061228900 x1610631750839424/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539208401 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 10 14:53:21 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 10 14:53:35 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539208408/real 1539208408] req@ffff880061228900 x1610631750839424/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539208415 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 10 14:53:35 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 10 14:53:56 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539208429/real 1539208429] req@ffff880061228900 x1610631750839424/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539208436 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 10 14:53:56 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 10 14:54:39 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539208471/real 1539208471] req@ffff880061228900 x1610631750839424/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539208478 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 10 14:54:39 oak-md1-s2 kernel: Lustre: 10546:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 10 14:54:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 10 14:54:46 oak-md1-s2 kernel: LustreError: 10546:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.17.26@o2ib6) returned error from blocking AST (req@ffff880061228900 x1610631750839424 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881af3f7ca00/0x6c00c408762867f5 lrc: 4/0,0 mode: PR/PR res: [0x200013483:0x19b76:0x0].0x0 bits 0x13 rrc: 85 type: IBT flags: 0x60200400000020 nid: 10.8.17.26@o2ib6 remote: 0xf1640f9a54fe78ba expref: 25 pid: 10532 timeout: 7492829018 lvb_type: 0 Oct 10 14:54:46 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.17.26@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 10 15:04:17 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 15:04:17 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 10 15:04:42 oak-md1-s2 kernel: LustreError: 10613:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.27.33@o2ib6) failed to reply to blocking AST (req@ffff881dde50b900 x1610631755313728 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff88176d454400/0x6c00c40881ae7f81 lrc: 1/0,0 mode: --/PR res: [0x200017f05:0x467a:0x0].0x0 bits 0x1b rrc: 5 type: IBT flags: 0x44a01400000020 nid: 10.8.27.33@o2ib6 remote: 0xc0bd4aa435f61242 expref: 40184 pid: 10613 timeout: 0 lvb_type: 0 Oct 10 15:04:42 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.27.33@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Oct 10 15:15:30 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 15:15:30 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Oct 10 15:23:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ac6a65d-520e-d82f-d7d1-98cbf5f62a0f (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880463a37800, cur 1539210182 expire 1539210032 last 1539209955 Oct 10 15:26:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 10 15:27:04 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 15:27:04 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 10 15:40:06 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 15:40:06 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 10 15:53:19 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 15:53:19 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 16:03:52 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 16:03:52 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages Oct 10 16:08:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9dd051b-4070-44e4-7155-3ef8aa21b480 (at 10.9.106.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac96b000, cur 1539212938 expire 1539212788 last 1539212711 Oct 10 16:16:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f77fd143-2b5d-a915-5503-29e45f2a4c71 (at 10.0.2.3@o2ib5) reconnecting Oct 10 16:16:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 10 16:17:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f77fd143-2b5d-a915-5503-29e45f2a4c71 (at 10.0.2.3@o2ib5) reconnecting Oct 10 16:17:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 10 16:17:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f77fd143-2b5d-a915-5503-29e45f2a4c71 (at 10.0.2.3@o2ib5) reconnecting Oct 10 16:17:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 10 16:19:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f77fd143-2b5d-a915-5503-29e45f2a4c71 (at 10.0.2.3@o2ib5) reconnecting Oct 10 16:19:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 10 16:24:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b7f67622-c1d7-8bfe-9858-d51809ea3543 (at 10.9.104.23@o2ib4) Oct 10 16:26:29 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 16:26:29 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 16:28:47 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 16:28:47 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 16:28:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c64361b9-320b-f5d0-8788-ff4905b98e4a (at 10.9.106.10@o2ib4) Oct 10 16:31:26 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 16:31:26 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 16:43:26 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 16:43:26 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 16:44:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de20dcbb-b59c-f703-8f5b-7296473e8494 (at 10.8.3.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bebfaa400, cur 1539215063 expire 1539214913 last 1539214836 Oct 10 16:45:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eaee074c-c42a-1b0d-d24f-6a4e0ab844fd (at 10.8.3.10@o2ib6) Oct 10 16:51:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e107773-36a2-de75-3f75-0afc7926b08b (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811eab21800, cur 1539215460 expire 1539215310 last 1539215233 Oct 10 16:52:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 16:53:55 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 16:53:55 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Oct 10 16:57:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ca0f3af-bdf9-2624-efcd-086021ed8ee8 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba2800, cur 1539215852 expire 1539215702 last 1539215625 Oct 10 16:58:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:02:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a292a2b5-784a-76cd-7134-cd4ad81f9ca2 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039f59f800, cur 1539216159 expire 1539216009 last 1539215932 Oct 10 17:03:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:07:29 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 17:07:29 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Oct 10 17:08:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1715946b-a2a0-44ab-01c1-96794f5093a9 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ed252000, cur 1539216496 expire 1539216346 last 1539216269 Oct 10 17:08:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:11:05 oak-md1-s2 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 0s: evicting client at 10.9.107.21@o2ib4 ns: mdt-oak-MDT0000_UUID lock: ffff881b6be95e00/0x6c00c4090b9ae165 lrc: 4/0,0 mode: PR/PR res: [0x200017cb7:0x25ed:0x0].0x0 bits 0x20 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.107.21@o2ib4 remote: 0x77884895719b8206 expref: 1553 pid: 10613 timeout: 0 lvb_type: 0 Oct 10 17:13:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c6a7ce7e-c04d-b634-7f4b-8901bb159ba6 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811eab26800, cur 1539216812 expire 1539216662 last 1539216585 Oct 10 17:14:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:18:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 61f8b31e-a5c9-7d33-b784-691f8e7349e4 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880858d69000, cur 1539217102 expire 1539216952 last 1539216875 Oct 10 17:19:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:21:19 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 17:21:19 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 10 17:23:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 998ba8ad-618d-9a21-e19c-377c3a117f2d (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103d67cc00, cur 1539217437 expire 1539217287 last 1539217210 Oct 10 17:24:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:28:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac584a02-4712-16c5-d493-d13445c96ee4 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881726b0fc00, cur 1539217733 expire 1539217583 last 1539217506 Oct 10 17:29:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:34:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cfe7304d-e7e4-290b-1844-48a08d02d38e (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cd009e400, cur 1539218066 expire 1539217916 last 1539217839 Oct 10 17:36:08 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 17:36:08 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 10 17:39:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 605767b2-8645-dab0-f287-fa5f89d12d77 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816bf254c00, cur 1539218392 expire 1539218242 last 1539218165 Oct 10 17:40:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:40:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 17:49:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0bbac41e-3891-7fed-1954-12e4b644f4e1 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac96c000, cur 1539218962 expire 1539218812 last 1539218735 Oct 10 17:49:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 17:50:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 17:50:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 17:50:53 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 17:50:53 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 18:00:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae7ec426-4df8-6585-faf4-f6d2064e70d2 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c31238400, cur 1539219636 expire 1539219486 last 1539219409 Oct 10 18:00:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 18:01:01 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 18:01:01 oak-md1-s2 kernel: Lustre: 10525:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 10 18:01:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 18:01:22 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 18:11:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a1d447d2-fbf4-f479-f700-ae46a93cfe7e (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881112f8d800, cur 1539220266 expire 1539220116 last 1539220039 Oct 10 18:11:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 18:11:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 18:11:48 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 18:13:31 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 18:13:31 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 10 18:22:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 18:22:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 18:24:18 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 18:24:18 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 18:27:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97cc1de6-0b9c-1c59-5709-47102274eb3a (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817a83aa000, cur 1539221223 expire 1539221073 last 1539220996 Oct 10 18:27:03 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 18:33:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 18:33:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 18:37:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 64cd55c4-35e5-ff58-380c-adf543501657 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7749400, cur 1539221874 expire 1539221724 last 1539221647 Oct 10 18:37:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 18:40:23 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 18:40:23 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Oct 10 18:44:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 18:44:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 18:48:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5f6cebc3-cd02-f52c-0a71-c1ef5a26c00b (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185e768000, cur 1539222494 expire 1539222344 last 1539222267 Oct 10 18:48:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 18:54:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 18:54:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 18:55:32 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 18:55:32 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 10 18:58:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97ef572b-fe42-0e26-86aa-326dc2698039 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e59b11800, cur 1539223109 expire 1539222959 last 1539222882 Oct 10 18:58:29 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 19:04:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 19:04:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 19:05:58 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 19:05:58 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 10 19:08:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 50e60db8-e2c9-1f81-6243-a6d26dc54126 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0da8d000, cur 1539223727 expire 1539223577 last 1539223500 Oct 10 19:08:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 19:14:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 19:14:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 19:19:28 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 19:19:28 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 19:19:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e2a09b1-3d2c-6078-e980-a3ef194bdf17 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef1800, cur 1539224371 expire 1539224221 last 1539224144 Oct 10 19:19:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 19:25:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 19:25:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 19:29:31 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 19:29:31 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 19:29:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ad7e0369-4df0-3f79-2ebc-b45317be1a56 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e6626000, cur 1539224994 expire 1539224844 last 1539224767 Oct 10 19:29:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 19:35:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 19:35:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 19:39:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 573eaf1a-8984-05fd-91af-c8b7ecd4d84e (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef4c00, cur 1539225595 expire 1539225445 last 1539225368 Oct 10 19:39:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 19:41:50 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 19:41:50 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 19:46:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 19:46:01 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 19:50:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 252fecf9-7269-36f1-4fc7-a2f70f2dc93f (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f366000, cur 1539226239 expire 1539226089 last 1539226012 Oct 10 19:50:39 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 19:55:05 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 19:55:05 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 10 19:56:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 19:56:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:01:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2baa78b0-630c-8e8b-ca4c-e2c4b8fc95b8 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bef20a400, cur 1539226884 expire 1539226734 last 1539226657 Oct 10 20:01:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:07:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 20:07:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:08:56 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 20:08:56 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Oct 10 20:11:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c292c59-ab06-6815-9bd4-a0536118e814 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881eccecfc00, cur 1539227495 expire 1539227345 last 1539227268 Oct 10 20:11:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:17:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 20:17:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:21:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0873916a-afb3-8b13-a299-6a99b6621b5b (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801393cd400, cur 1539228106 expire 1539227956 last 1539227879 Oct 10 20:21:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:26:41 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 20:26:41 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 20:27:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 20:27:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:32:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 66a69096-d97c-6928-a703-6654836628be (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880be64c5400, cur 1539228723 expire 1539228573 last 1539228496 Oct 10 20:32:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 20:37:10 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 20:37:10 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 20:38:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 20:38:13 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 20:42:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0427745a-709c-a70a-8938-c47fac7c6f3f (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880185a68c00, cur 1539229346 expire 1539229196 last 1539229119 Oct 10 20:42:26 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 20:47:32 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 20:47:32 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 10 20:48:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 20:48:24 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 20:52:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59d930df-783d-c21c-2bcb-3f26e36a4631 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc080c400, cur 1539229956 expire 1539229806 last 1539229729 Oct 10 20:52:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 20:57:44 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 20:57:44 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Oct 10 20:58:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 20:58:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 21:02:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb9f8373-315e-32b6-f520-2bbd0933cf51 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dfc5b0400, cur 1539230576 expire 1539230426 last 1539230349 Oct 10 21:02:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 21:09:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 21:09:04 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 21:13:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5fe7e5c7-a7be-0093-94e2-fe5439e88f32 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d74ea1c00, cur 1539231197 expire 1539231047 last 1539230970 Oct 10 21:13:17 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 21:14:55 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 21:14:55 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 21:19:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 21:19:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 21:23:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1932254b-3612-102b-8521-2cc3bb082708 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88201677c000, cur 1539231821 expire 1539231671 last 1539231594 Oct 10 21:23:41 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 21:30:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 21:30:07 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 21:33:57 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 21:33:57 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 21:34:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client afd291f3-c4f0-f9c9-0876-e295a765670e (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb981f400, cur 1539232459 expire 1539232309 last 1539232232 Oct 10 21:34:19 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 21:40:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 21:40:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 21:50:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cbb35f51-b9da-10ed-263f-87c119c10be5 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da7a77c00, cur 1539233400 expire 1539233250 last 1539233173 Oct 10 21:50:00 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 21:51:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 21:51:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 22:00:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2064c287-d13e-75e2-9f26-33c6771d9c48 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac224400, cur 1539234042 expire 1539233892 last 1539233815 Oct 10 22:00:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 22:01:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 22:01:25 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 22:04:16 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 22:04:16 oak-md1-s2 kernel: Lustre: 10631:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 22:11:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 22:11:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 22:12:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0d36cbc8-faf4-c808-8c7e-3d85aa6b7863 (at 10.12.4.72@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800585fa400, cur 1539234776 expire 1539234626 last 1539234549 Oct 10 22:12:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 22:22:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 22:22:15 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 22:24:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dbeb2df9-d26d-27b7-f9f2-de052d2b55e9 (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801919dd000, cur 1539235449 expire 1539235299 last 1539235222 Oct 10 22:24:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 22:32:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 22:32:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 22:36:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7ac8de2e-8ab4-3b5d-9840-f8ce97a2333c (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817152e7800, cur 1539236199 expire 1539236049 last 1539235972 Oct 10 22:36:39 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 22:42:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 22:42:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 10 22:47:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cfea22ab-958e-970e-afc2-d3035e14e54b (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e87510000, cur 1539236822 expire 1539236672 last 1539236595 Oct 10 22:47:02 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 22:53:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 22:53:03 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 22:53:34 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 22:53:34 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 10 22:57:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b56b3911-3610-a216-69aa-085f3817311f (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88127d2e0c00, cur 1539237435 expire 1539237285 last 1539237208 Oct 10 22:57:15 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 23:03:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 10 23:03:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 23:03:55 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:03:55 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 10 23:07:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b55f3df-a8a2-1391-7419-ec3ecd4ad34f (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc48ccc00, cur 1539238067 expire 1539237917 last 1539237840 Oct 10 23:07:47 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 23:08:54 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:08:54 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 23:10:06 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:10:06 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 10 23:13:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 23:13:49 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 10 23:15:05 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:15:05 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 23:15:53 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:15:53 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 10 23:18:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 63fcf484-1aa5-70f5-e9e7-683ce50a7a50 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f4c00, cur 1539238681 expire 1539238531 last 1539238454 Oct 10 23:18:01 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 23:20:57 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:20:57 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 10 23:24:02 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:24:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 23:24:12 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 23:26:29 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:26:29 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 10 23:28:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 23ab9aba-0214-e613-8e09-e8aa2b5d7243 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beea0ac00, cur 1539239305 expire 1539239155 last 1539239078 Oct 10 23:28:25 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 23:30:01 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:30:01 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 10 23:34:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 23:34:39 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 10 23:38:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb0ac66c-6dcd-de2f-050e-3fe340f32495 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881888aa6800, cur 1539239931 expire 1539239781 last 1539239704 Oct 10 23:38:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 10 23:39:15 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:39:15 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 10 23:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 10 23:44:49 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 23:46:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 10 23:49:50 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:49:50 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 10 23:50:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b37d64fb-3c48-bc72-ddb2-538fa9a188df (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210d800, cur 1539240626 expire 1539240476 last 1539240399 Oct 10 23:50:26 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 10 23:59:55 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 10 23:59:55 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 40 previous similar messages Oct 11 00:10:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f391cb18-2dea-a403-02b9-7c06005e60c4 (at 10.9.102.25@o2ib4) Oct 11 00:10:59 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 00:11:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 57ec56d4-1b1a-3e5e-8faa-8f4bfba3857a (at 10.9.102.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040e08000, cur 1539241914 expire 1539241764 last 1539241687 Oct 11 00:11:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 00:16:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 00:16:33 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 00:16:33 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 11 00:16:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f5e8306c-7a6a-be9a-8756-603f99368461 (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db00d3400, cur 1539242204 expire 1539242054 last 1539241977 Oct 11 00:28:05 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 00:28:05 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 11 00:39:11 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 00:39:11 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 11 00:49:45 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 00:49:45 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 31 previous similar messages Oct 11 00:59:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ba7efc4-2905-7880-373c-99d4982d095c (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe1343c00, cur 1539244784 expire 1539244634 last 1539244557 Oct 11 00:59:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 01:00:54 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 01:00:54 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Oct 11 01:11:31 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 01:11:31 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Oct 11 01:22:00 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 01:22:00 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 25 previous similar messages Oct 11 01:32:54 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 01:32:54 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 01:44:31 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 01:44:31 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Oct 11 01:55:46 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 01:55:46 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 11 02:07:33 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 02:07:33 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 11 02:25:25 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 02:25:25 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14 previous similar messages Oct 11 02:44:15 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 02:44:15 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Oct 11 02:48:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 02:48:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 02:49:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 985b4f1e-45c0-2994-9c26-724067dc38bb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88166b2e8c00, cur 1539251370 expire 1539251220 last 1539251143 Oct 11 02:54:24 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 02:54:24 oak-md1-s2 kernel: Lustre: 10634:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 11 02:57:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 02:58:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ece6d706-77ab-8624-3b88-f9b8d65853b1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808cdbdd400, cur 1539251904 expire 1539251754 last 1539251677 Oct 11 03:05:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e4c7287-248d-b244-9614-2fdbfc272305 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a1ee35800, cur 1539252339 expire 1539252189 last 1539252112 Oct 11 03:06:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 03:07:14 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 03:07:14 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 11 03:08:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 11 03:09:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d2c7bc75-a482-623e-76e9-f285f04a2755 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099ad22000, cur 1539252575 expire 1539252425 last 1539252348 Oct 11 03:14:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1bad6f10-4e6d-3737-2cc2-89c6951fbb11 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ba2a67400, cur 1539252899 expire 1539252749 last 1539252672 Oct 11 03:16:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 03:17:16 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 03:17:16 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 03:25:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eafe77e3-8223-6c72-d95a-69ac2c1a15b6 (at 10.12.4.38@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e71cd400, cur 1539253526 expire 1539253376 last 1539253299 Oct 11 03:27:27 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 03:27:27 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17 previous similar messages Oct 11 03:34:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 03:35:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7c34b2ea-ab1c-62f2-fab5-835e2145e070 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881875ecd400, cur 1539254114 expire 1539253964 last 1539253887 Oct 11 03:38:52 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 03:38:52 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 11 03:46:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 943bd0c0-3ccc-538a-8bc9-8b3939e62b3b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881868a0ec00, cur 1539254800 expire 1539254650 last 1539254573 Oct 11 03:48:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 03:52:34 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 03:52:34 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 11 04:03:03 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 04:03:03 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 11 04:14:40 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 04:14:40 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages Oct 11 04:25:14 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 04:25:14 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages Oct 11 04:38:22 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 04:38:22 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 11 04:45:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 11 04:46:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 199f7ff9-d1cc-011b-2c84-0a121cfb028a (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88077a786000, cur 1539258360 expire 1539258210 last 1539258133 Oct 11 04:49:40 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 04:49:40 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Oct 11 05:00:38 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 05:00:38 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 32 previous similar messages Oct 11 05:11:19 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 05:11:19 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 28 previous similar messages Oct 11 05:11:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6dacf764-0fc0-fc80-1858-6bc1432319f5 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567b9000, cur 1539259916 expire 1539259766 last 1539259689 Oct 11 05:12:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 05:16:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4f03101-1d70-bef1-3cf9-ecae7018dcd0 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101cff1800, cur 1539260211 expire 1539260061 last 1539259984 Oct 11 05:17:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 05:22:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a7c27916-56bc-f8fc-6133-4459103edb82 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f19685c00, cur 1539260524 expire 1539260374 last 1539260297 Oct 11 05:22:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 05:25:16 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 05:25:16 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 54 previous similar messages Oct 11 05:26:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 05:26:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae8c2be2-abe6-86b6-99cd-640d007493bd (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881beea6d000, cur 1539260808 expire 1539260658 last 1539260581 Oct 11 05:28:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 05:38:13 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 05:38:13 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Oct 11 05:49:59 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 05:49:59 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Oct 11 05:51:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 726a0ae2-ac16-6786-a632-fa699e6337b2 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880761e36000, cur 1539262266 expire 1539262116 last 1539262039 Oct 11 05:51:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 05:51:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 05:56:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f35ba76-da11-e0b1-e03a-d5ce241f7ad4 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880036e53c00, cur 1539262561 expire 1539262411 last 1539262334 Oct 11 05:56:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:01:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f112de6-34fc-66fd-7373-41872b1e2083 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf36d2c00, cur 1539262889 expire 1539262739 last 1539262662 Oct 11 06:02:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:05:06 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 06:05:06 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 11 06:06:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5beb0add-ca66-1f3e-88ca-8a5f48ea677f (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eda28f000, cur 1539263184 expire 1539263034 last 1539262957 Oct 11 06:06:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 11 06:07:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:11:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec0e8de9-c62c-46b4-c407-23e26ef70567 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d22c00, cur 1539263488 expire 1539263338 last 1539263261 Oct 11 06:11:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 06:11:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 11 06:12:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:14:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 11 06:15:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8268057e-5149-1ffd-aff9-4c4c24ac9dcb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227cf400, cur 1539263726 expire 1539263576 last 1539263499 Oct 11 06:15:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 06:15:59 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 06:15:59 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 11 06:17:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:21:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d4de6170-580b-c998-3556-ef77df4645fe (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ac06fa400, cur 1539264102 expire 1539263952 last 1539263875 Oct 11 06:21:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 06:22:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:26:44 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 06:26:44 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 32 previous similar messages Oct 11 06:32:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 793b89a3-c45f-a939-f0ba-57de9a9f1420 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88128aa01000, cur 1539264736 expire 1539264586 last 1539264509 Oct 11 06:32:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 06:33:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:33:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 06:39:01 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 06:39:01 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 33 previous similar messages Oct 11 06:42:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0dcd532-7688-87c1-ed61-cfa24d026c5c (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e6ba48000, cur 1539265343 expire 1539265193 last 1539265116 Oct 11 06:42:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 06:43:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:43:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 06:49:58 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 06:49:58 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 42 previous similar messages Oct 11 06:53:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 750d08d8-a7af-8f88-4b9e-a39dab9ee528 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ecb32400, cur 1539265987 expire 1539265837 last 1539265760 Oct 11 06:53:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 06:53:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 06:53:48 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 07:03:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d455b4c-e36f-1e56-79f8-3f367f6cae21 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9e400, cur 1539266597 expire 1539266447 last 1539266370 Oct 11 07:03:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 07:04:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 07:04:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 07:08:42 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 07:08:42 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 33 previous similar messages Oct 11 07:14:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49a6439d-9217-d68c-0b1d-1daf2717e70f (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882006163400, cur 1539267241 expire 1539267091 last 1539267014 Oct 11 07:14:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 07:14:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 07:14:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 07:24:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f45d5f3-7aa5-f98c-039d-2e17ba87394a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818987dd400, cur 1539267848 expire 1539267698 last 1539267621 Oct 11 07:24:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 07:25:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 07:25:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 07:27:24 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 07:27:24 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 07:34:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 417d3713-2edd-02ef-6625-1949a4d3010b (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881691145c00, cur 1539268470 expire 1539268320 last 1539268243 Oct 11 07:34:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 07:35:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 07:35:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 07:42:50 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 07:42:50 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 07:44:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b3c8cac3-8ec9-bdcb-8c54-a6555d8686a7 (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881906264000, cur 1539269098 expire 1539268948 last 1539268871 Oct 11 07:44:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 07:56:00 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 07:56:00 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 11 08:11:51 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 08:26:31 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 08:26:31 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 11 08:30:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 11 08:30:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 08:30:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b648e03-5493-bdbe-a9f3-c7979832cfc6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e0800, cur 1539271827 expire 1539271677 last 1539271600 Oct 11 08:34:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd6af681-8c66-840b-72b3-8731690e3ca8 (at 10.8.10.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef2884400, cur 1539272059 expire 1539271909 last 1539271832 Oct 11 08:35:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f0271459-b031-b436-9d25-f946c84b4a1a (at 10.8.10.12@o2ib6) Oct 11 08:41:09 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 08:41:09 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 08:53:27 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 08:53:27 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 09:04:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 11 09:05:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Oct 11 09:07:58 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 09:07:58 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 09:08:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9eaab7d-29ff-bb62-a352-bbec77c9330c (at 10.8.2.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc080a400, cur 1539274118 expire 1539273968 last 1539273891 Oct 11 09:25:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 11 09:25:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 09:26:41 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 09:26:41 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 09:34:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1585ed96-ba3a-af1b-3b88-056c0a1fcdb8 (at 10.8.2.2@o2ib6) Oct 11 09:35:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f8742cd5-0696-73da-734e-8b901d11dea0 (at 10.8.17.29@o2ib6) Oct 11 09:36:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 912b02a7-46b6-d271-4940-efccd23bb47a (at 10.8.2.10@o2ib6) Oct 11 09:45:24 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 09:45:24 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 24 previous similar messages Oct 11 09:55:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 11 09:59:10 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 09:59:10 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 10:02:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d0e21c38-c8ae-043d-32e8-b073fbe90929 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809bd80c800, cur 1539277324 expire 1539277174 last 1539277097 Oct 11 10:02:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 10:04:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 11 10:18:22 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 10:18:22 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 11 10:24:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9fd0388-5ea8-ff7e-0d50-8dc8bbd79efd (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac42a000, cur 1539278666 expire 1539278516 last 1539278439 Oct 11 10:26:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 11 10:33:26 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 10:33:26 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 10:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bd1333d-8fa3-e1a3-e560-c3942c19d327 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582305400, cur 1539279287 expire 1539279137 last 1539279060 Oct 11 10:34:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 11 10:50:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d934f978-4f57-07ff-2834-17ed679f5151 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881eccece400, cur 1539280235 expire 1539280085 last 1539280008 Oct 11 10:51:59 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 10:51:59 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 11 10:55:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ece9c448-e806-1c08-29cb-3978e1671c0f (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe5a5dc00, cur 1539280538 expire 1539280388 last 1539280311 Oct 11 10:56:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 11:07:07 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 11:07:07 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 11 11:14:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ed662e33-0383-69b6-3f65-54f4b5c84e64 (at 10.9.107.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061e804c00, cur 1539281650 expire 1539281500 last 1539281423 Oct 11 11:16:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 11:17:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2773cb2-f9d0-80e3-1b17-bb2f832cc646 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881034b0a800, cur 1539281853 expire 1539281703 last 1539281626 Oct 11 11:18:12 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 11:18:12 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 11:20:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 11 11:21:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 85abc303-5f5a-116e-4fc1-1bb592e4536c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e2205d000, cur 1539282071 expire 1539281921 last 1539281844 Oct 11 11:25:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e5886901-6e65-827c-9dac-ad86b9953379 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881112f8a400, cur 1539282320 expire 1539282170 last 1539282093 Oct 11 11:26:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 11:29:27 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 11:29:27 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 11 11:34:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 11:35:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c602cf04-7d3b-f122-5fe9-f99bbd9ea210 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c27d2c00, cur 1539282904 expire 1539282754 last 1539282677 Oct 11 11:39:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d9a3bb5b-fd3f-ae39-4c27-9417a067c280 (at 10.9.106.9@o2ib4) Oct 11 11:43:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 11:43:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a8c9916-098a-f083-141f-af8262be5d25 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1f6a9000, cur 1539283432 expire 1539283282 last 1539283205 Oct 11 11:48:15 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 11:49:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e76d7a93-55b4-1e86-6751-3a9e278ca5ba (at 10.9.101.57@o2ib4) Oct 11 11:53:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 11:53:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55e59d76-5a9c-c059-9500-a6cbfe347b04 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805c27d2c00, cur 1539284019 expire 1539283869 last 1539283792 Oct 11 12:00:40 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 12:00:40 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 11 12:03:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 12:03:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf1164e1-aeec-ef89-4d94-6a88b76d9d7a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b42aa2800, cur 1539284622 expire 1539284472 last 1539284395 Oct 11 12:13:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 12:14:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8c1b2b2e-ad1a-4e6c-23fa-4066840047ba (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0315b000, cur 1539285245 expire 1539285095 last 1539285018 Oct 11 12:14:59 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 12:14:59 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 11 12:24:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6db18ef-e70c-812a-f76e-f7a1e6bf56be (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfefc0800, cur 1539285870 expire 1539285720 last 1539285643 Oct 11 12:25:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 12:32:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d1975b31-6866-6672-846e-417fac9b4d24 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6d1d9000, cur 1539286337 expire 1539286187 last 1539286110 Oct 11 12:33:36 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 12:33:36 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 11 12:34:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 12:34:35 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 12:45:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 12:45:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d393d71f-0f94-d187-73be-e8ffd6e0a0e0 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d38d0400, cur 1539287127 expire 1539286977 last 1539286900 Oct 11 12:45:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 12:49:30 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 12:49:30 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 11 12:56:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 379f0f9e-2aeb-5022-2eda-d2ec16c392a9 (at 10.8.22.25@o2ib6) reconnecting Oct 11 12:56:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a4d6ebc4-1eae-1df9-1e22-87ae53fd9c31 (at 10.8.22.25@o2ib6) Oct 11 13:00:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43313e8c-677e-f31e-d33d-8e6ef05ebda3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b68e2c00, cur 1539288052 expire 1539287902 last 1539287825 Oct 11 13:03:31 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 13:15:14 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 13:24:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 13:24:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 13:25:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0f7fd43-4719-3e7d-ec83-217032c85b87 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014e258c00, cur 1539289504 expire 1539289354 last 1539289277 Oct 11 13:26:31 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 13:26:31 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 11 13:27:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 13:28:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1d7b0716-4530-43cf-1bda-ab639301e922 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb2138800, cur 1539289713 expire 1539289563 last 1539289486 Oct 11 13:30:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 13:31:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 220afc2c-bb26-eeff-32c4-56a4e696fc01 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882006161000, cur 1539289868 expire 1539289718 last 1539289641 Oct 11 13:37:10 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 13:37:10 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 13:38:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 13:38:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 13:39:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 621ee118-f041-798d-aa58-b5ff7f8d3723 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818973fcc00, cur 1539290372 expire 1539290222 last 1539290145 Oct 11 13:39:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 13:51:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 13:51:22 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 13:51:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb129f16-0be2-d77e-6fd7-3384d2003af6 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808cf36d400, cur 1539291117 expire 1539290967 last 1539290890 Oct 11 13:51:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 13:54:00 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 13:54:00 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 14:04:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 14:04:18 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 14:04:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a842300d-e875-0957-7a70-5aaabcf64401 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88172937cc00, cur 1539291897 expire 1539291747 last 1539291670 Oct 11 14:04:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 14:07:19 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 14:07:19 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Oct 11 14:18:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 14:18:41 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 14:19:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae89e891-0d89-c932-46da-2f63c2ff499e (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c29680000, cur 1539292794 expire 1539292644 last 1539292567 Oct 11 14:19:54 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 14:25:13 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 14:25:13 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 14:29:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 14:29:51 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 14:30:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 83fbcdf8-60de-295c-9a52-c478606a57fe (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f360000, cur 1539293409 expire 1539293259 last 1539293182 Oct 11 14:30:09 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 14:36:35 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 14:36:35 oak-md1-s2 kernel: Lustre: 10513:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 11 14:40:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b8184b9-046d-b888-0b85-576f45e9375c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88130474e800, cur 1539294018 expire 1539293868 last 1539293791 Oct 11 14:40:18 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 14:42:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 14:42:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 14:51:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b79a0032-fe0a-39a6-eb90-bc353ef9b5fd (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040ed4800, cur 1539294713 expire 1539294563 last 1539294486 Oct 11 14:51:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 14:52:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 14:58:04 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 14:58:04 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 11 14:59:32 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 14:59:32 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 11 15:01:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 696f1799-1aed-b45f-b199-1a129a126a1a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101be04c00, cur 1539295314 expire 1539295164 last 1539295087 Oct 11 15:01:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 11 15:03:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 15:03:04 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 15:06:56 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 15:06:56 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 11 15:13:34 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 15:13:34 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 11 15:20:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c97ac67f-5e6c-52df-e491-cb64fb4b0720 (at 10.8.8.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039237800, cur 1539296413 expire 1539296263 last 1539296186 Oct 11 15:20:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 11 15:24:17 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 15:24:17 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 41 previous similar messages Oct 11 15:31:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 11 15:31:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 203afd7f-4869-c272-090f-31192c088b07 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daaa90c00, cur 1539297111 expire 1539296961 last 1539296884 Oct 11 15:35:16 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 15:35:16 oak-md1-s2 kernel: Lustre: 4694:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17 previous similar messages Oct 11 15:41:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3427dd05-b49e-520d-ab57-b133ca713edf (at 10.8.8.32@o2ib6) Oct 11 15:45:53 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 15:45:53 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 11 16:15:15 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 16:15:15 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Oct 11 16:24:03 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 16:24:03 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 16:26:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4d201342-6116-5755-dae2-c3d379c57105 (at 10.9.104.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef2886400, cur 1539300366 expire 1539300216 last 1539300139 Oct 11 16:38:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 16:39:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47e21359-077b-aa71-dd86-6bacf6072f5d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c6932fc00, cur 1539301140 expire 1539300990 last 1539300913 Oct 11 16:45:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 16:46:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b8874aaf-3b05-d06b-0d83-f2f280b4720b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ed2220c00, cur 1539301570 expire 1539301420 last 1539301343 Oct 11 16:48:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4d201342-6116-5755-dae2-c3d379c57105 (at 10.9.104.45@o2ib4) Oct 11 16:53:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 22a0d089-4baf-2f22-e1c6-2f7eead5de80 (at 10.9.106.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88197aa5f400, cur 1539302001 expire 1539301851 last 1539301774 Oct 11 16:53:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 16:58:45 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:01:37 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:03:23 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:03:23 oak-md1-s2 kernel: Lustre: 7070:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 11 17:03:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 11 17:04:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 60e08d72-3a35-edf9-2a42-8d1b0bd42fc1 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185baccc00, cur 1539302661 expire 1539302511 last 1539302434 Oct 11 17:04:21 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 11 17:14:05 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:16:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5d71c446-ea78-efee-66e1-229a2ce489dc (at 10.9.104.66@o2ib4) Oct 11 17:16:54 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:16:54 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 11 17:24:35 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:24:35 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 11 17:36:11 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:36:11 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 28 previous similar messages Oct 11 17:47:29 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:47:29 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 11 17:58:39 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 17:58:39 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 11 18:08:43 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 18:08:43 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 11 18:11:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 60229b59-2f5d-a32f-9144-58da0612a678 (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814f0362800, cur 1539306665 expire 1539306515 last 1539306438 Oct 11 18:24:01 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 18:24:01 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 11 18:25:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 11 18:25:03 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 11 18:34:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 24d983c1-d007-5722-9121-50e2be5d8758 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa870400, cur 1539308061 expire 1539307911 last 1539307834 Oct 11 18:35:07 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 18:35:07 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 54 previous similar messages Oct 11 18:36:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 18:45:16 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 18:45:16 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 87 previous similar messages Oct 11 18:46:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 18:46:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47ee19b7-f8bd-8d50-2622-e9f21fba7753 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102abea000, cur 1539308790 expire 1539308640 last 1539308563 Oct 11 18:54:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 18:54:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a9f55aa-c04c-362f-3fe0-6847ae239255 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0a6f8000, cur 1539309287 expire 1539309137 last 1539309060 Oct 11 18:55:24 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 18:55:24 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 79 previous similar messages Oct 11 19:03:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c9b9e3a2-eff0-07fe-a3f4-0a7711ee2294 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880877ce8400, cur 1539309800 expire 1539309650 last 1539309573 Oct 11 19:05:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 19:05:59 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 19:05:59 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 117 previous similar messages Oct 11 19:16:25 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 19:16:25 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 121 previous similar messages Oct 11 19:27:32 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 19:27:32 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 51 previous similar messages Oct 11 19:37:32 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 19:37:32 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Oct 11 19:48:19 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 19:48:19 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 251 previous similar messages Oct 11 19:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 19:59:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 89cd2bfc-c0af-8de1-a68c-748289212ec9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061e806c00, cur 1539313159 expire 1539313009 last 1539312932 Oct 11 19:59:41 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 19:59:41 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 174 previous similar messages Oct 11 20:09:50 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 20:09:50 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 40 previous similar messages Oct 11 20:20:16 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 20:20:16 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 71 previous similar messages Oct 11 20:34:18 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 20:34:18 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 59 previous similar messages Oct 11 20:47:49 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 20:47:49 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 100 previous similar messages Oct 11 20:58:57 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 20:58:57 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 11 21:04:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 27080df9-909d-bf43-957d-45b0f380145f (at 10.9.106.27@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880be64c3c00, cur 1539317092 expire 1539316942 last 1539316865 Oct 11 21:09:10 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 21:09:10 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 11 21:19:19 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 21:19:19 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Oct 11 21:29:41 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 21:29:41 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Oct 11 21:30:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9ebd175-47c5-a73c-a10b-9bb5e732c7b8 (at 10.9.106.27@o2ib4) Oct 11 21:40:04 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 21:40:04 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages Oct 11 21:52:15 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 21:52:15 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 11 22:02:16 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 22:02:16 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 11 22:12:30 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 22:12:30 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 58 previous similar messages Oct 11 22:25:17 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 22:25:17 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 11 22:36:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 11 22:37:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c0398846-8084-7fe9-79e5-c54cfb045aaa (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014e258000, cur 1539322639 expire 1539322489 last 1539322412 Oct 11 22:42:53 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 22:42:53 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Oct 11 22:50:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ff920c92-8813-af7a-233e-9c6f1f6b493c (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800496b5800, cur 1539323428 expire 1539323278 last 1539323201 Oct 11 22:56:42 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 22:56:42 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 11 23:13:20 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 23:13:20 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 11 23:27:00 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 23:27:00 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 11 23:37:39 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 23:37:39 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 11 23:55:25 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 11 23:55:25 oak-md1-s2 kernel: Lustre: 10585:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 12 00:07:00 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 00:07:00 oak-md1-s2 kernel: Lustre: 10515:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 00:18:38 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 00:18:38 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 12 00:30:52 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 00:30:52 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 23 previous similar messages Oct 12 00:34:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1dd8a67d-b700-09e9-3e9c-b339c4274932 (at 10.12.4.23@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da97ec800, cur 1539329648 expire 1539329498 last 1539329421 Oct 12 00:43:43 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 00:43:43 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 12 01:59:37 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 01:59:37 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 12 02:01:55 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 02:01:55 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 12 02:07:07 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 02:21:02 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 02:50:34 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 02:50:34 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 02:57:45 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 02:57:45 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 03:07:09 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 03:15:16 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 03:15:16 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 12 03:40:49 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 03:40:49 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 64 previous similar messages Oct 12 03:42:56 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:05:46 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:05:46 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 04:10:38 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:10:38 oak-md1-s2 kernel: Lustre: 10581:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 12 04:13:07 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:36:10 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:36:10 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 04:36:28 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:36:28 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 12 04:46:03 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:46:03 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 04:48:37 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:52:17 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 04:52:17 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 04:57:53 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 05:04:15 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 05:04:15 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 12 05:14:39 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 05:14:39 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 12 05:29:19 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 05:29:19 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 32 previous similar messages Oct 12 05:44:54 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 05:44:54 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Oct 12 05:54:57 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 05:54:57 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 12 06:07:49 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 06:07:49 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 06:19:56 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 06:19:56 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 23 previous similar messages Oct 12 06:39:26 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 06:39:26 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 122 previous similar messages Oct 12 06:51:28 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 06:51:28 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 12 07:02:19 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 07:02:19 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 07:12:21 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 07:12:21 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 17 previous similar messages Oct 12 07:24:08 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 07:24:08 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 12 07:36:03 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 07:36:03 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 12 07:50:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fbb83184-3e78-df7b-5109-069e289706b1 (at 10.9.101.54@o2ib4) Oct 12 07:50:53 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 07:50:53 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 12 07:51:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2944f2f1-02de-259a-b62f-645494d44cf8 (at 10.9.101.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacc5b000, cur 1539355915 expire 1539355765 last 1539355688 Oct 12 08:06:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 410025aa-6f42-c42b-9e79-bce2018737f4 (at 10.12.4.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103da72000, cur 1539356801 expire 1539356651 last 1539356574 Oct 12 08:07:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 410025aa-6f42-c42b-9e79-bce2018737f4 (at 10.12.4.67@o2ib) Oct 12 08:07:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f97f5c4e-ea1b-a56f-2caf-9e8a029f9f0e (at 10.12.4.30@o2ib) Oct 12 08:07:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to edf32d0b-88eb-f1d4-00b8-c0af2b7893e4 (at 10.12.4.38@o2ib) Oct 12 08:07:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e37fecd3-1f0f-58f5-2d3b-ef0ed4e1a65f (at 10.12.4.41@o2ib) Oct 12 08:08:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0c7eefa3-fc99-540c-06e7-6b75b4a7596a (at 10.12.4.71@o2ib) Oct 12 08:12:06 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 08:22:22 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 08:22:22 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 08:26:01 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 08:26:01 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 12 08:31:28 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 08:31:28 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 08:39:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 78708d32-8777-a39c-9631-b44481ea13c3 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8af9000, cur 1539358762 expire 1539358612 last 1539358535 Oct 12 08:40:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Oct 12 08:45:34 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 08:45:34 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 12 09:06:04 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 09:06:04 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 09:09:07 oak-md1-s2 kernel: Lustre: 3869:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 09:09:07 oak-md1-s2 kernel: Lustre: 3869:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 09:14:25 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 09:14:25 oak-md1-s2 kernel: Lustre: 4546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 09:22:28 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 09:22:28 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 09:26:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ffe30d3-8e78-711d-0c19-038a46a3db73 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103abb4c00, cur 1539361564 expire 1539361414 last 1539361337 Oct 12 09:27:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Oct 12 09:36:33 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 09:36:33 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 09:47:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 09:48:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9796be16-1130-7d07-9e43-6ba000d75821 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b268d000, cur 1539362919 expire 1539362769 last 1539362692 Oct 12 09:51:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f5344e94-a8d2-655c-95ff-511fa32d44d5 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0da8c000, cur 1539363101 expire 1539362951 last 1539362874 Oct 12 09:58:03 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 09:58:03 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 10:03:08 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 10:03:08 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 10:06:25 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 10:06:25 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 10:12:19 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 10:12:19 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 10:18:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 10:18:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30e6b989-8359-c3ff-480b-47a01c8d33a6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93136400, cur 1539364727 expire 1539364577 last 1539364500 Oct 12 10:26:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 10:26:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 56d1aa41-9985-2779-7f11-98bc20c8129a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f712800, cur 1539365218 expire 1539365068 last 1539364991 Oct 12 10:33:00 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 10:33:00 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages Oct 12 10:34:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 10:34:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ca895adc-ab81-fe91-16df-1873f6b3b7b3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b7da000, cur 1539365696 expire 1539365546 last 1539365469 Oct 12 10:41:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 10:42:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eee8bc2c-04d8-0dfa-c128-e72743eba734 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf015000, cur 1539366121 expire 1539365971 last 1539365894 Oct 12 10:46:52 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 10:46:52 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 10:49:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 10:50:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b93d4fa1-0b12-c50e-799b-ee44b26ec20f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b275af400, cur 1539366616 expire 1539366466 last 1539366389 Oct 12 10:57:24 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 10:58:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 10:58:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 07dcc37d-b730-f069-4ed7-b5135a18b515 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881751192000, cur 1539367101 expire 1539366951 last 1539366874 Oct 12 11:03:23 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:03:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d52f0091-3a27-7e42-d6d4-30f7762ea827 (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880761e31000, cur 1539367428 expire 1539367278 last 1539367201 Oct 12 11:06:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 11:07:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e60161d-ffc4-162a-2008-c6338a99e751 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880343110000, cur 1539367636 expire 1539367486 last 1539367409 Oct 12 11:07:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 12 11:08:42 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:09:18 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:09:18 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 11:09:33 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:09:55 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:09:55 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 11:10:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 12 11:11:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0059980b-89de-a75b-4923-c32e13fdb562 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af4800, cur 1539367884 expire 1539367734 last 1539367657 Oct 12 11:11:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 12 11:12:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b7f67622-c1d7-8bfe-9858-d51809ea3543 (at 10.9.104.23@o2ib4) Oct 12 11:12:52 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:12:52 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 11:15:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 11:15:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bce03b07-bed3-c116-2ec7-2823e38857c9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881002614000, cur 1539368112 expire 1539367962 last 1539367885 Oct 12 11:15:30 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:15:30 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 11:18:44 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:18:44 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 11:23:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 11:23:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 22ac6984-b4dc-ce34-62f7-51989277b07d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e383c00, cur 1539368630 expire 1539368480 last 1539368403 Oct 12 11:29:22 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:29:22 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 11:35:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 11:35:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 12 11:36:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 139ddbaa-443d-c895-3931-fd04bfca632a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200bed6400, cur 1539369366 expire 1539369216 last 1539369139 Oct 12 11:40:12 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:40:12 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 12 11:49:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 11:49:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eee0084f-c93f-8f95-202d-3e6d1f623221 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c64a84c00, cur 1539370182 expire 1539370032 last 1539369955 Oct 12 11:53:35 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 11:53:35 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 12 12:01:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 12:01:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 12 12:02:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aed6b9fd-dd43-8371-bdf2-e126a0ff9060 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88085dc1bc00, cur 1539370954 expire 1539370804 last 1539370727 Oct 12 12:02:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 12 12:13:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 12:14:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f67e702-3fe1-955f-97b9-ca94dedf15e7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d14649000, cur 1539371692 expire 1539371542 last 1539371465 Oct 12 12:20:35 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 12:20:35 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Oct 12 12:23:10 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 12:26:11 oak-md1-s2 kernel: LNet: 206745:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.245@o2ib5 version 12/12 incarnation 1538524503468256/1539372370411081 Oct 12 12:26:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fccddc08-d568-f4b4-6a44-f4e21b04be63 (at 10.0.2.245@o2ib5) Oct 12 12:28:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aa357186-ec09-c1ab-d553-fc19bb820938 (at 10.0.2.245@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e8d9b9000, cur 1539372532 expire 1539372382 last 1539372305 Oct 12 12:30:43 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 12:30:43 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 12 12:44:58 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 12:57:58 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 12:57:58 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 12 13:01:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59513c82-681e-a79a-2773-04b0c2bddfb8 (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880038e8b800, cur 1539374519 expire 1539374369 last 1539374292 Oct 12 13:16:21 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 13:16:21 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 13:17:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 12 13:18:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bd32a4c7-49e6-d4d7-fd1f-8f9c16adf00f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e75233400, cur 1539375491 expire 1539375341 last 1539375264 Oct 12 13:19:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f74eeb0-8bc6-7d26-2bd9-23a8e9676011 (at 10.12.4.25@o2ib) in 171 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e4c00, cur 1539375567 expire 1539375417 last 1539375396 Oct 12 13:29:00 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 13:29:00 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 13:34:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 12 13:34:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7920d0fc-5917-dba7-b142-8d3ba7d377eb (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881734bc9000, cur 1539376484 expire 1539376334 last 1539376257 Oct 12 13:42:50 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 13:48:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 39d891b9-c4f4-dbea-4a6b-207a63490d00 (at 10.9.107.22@o2ib4) Oct 12 13:53:43 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 13:53:43 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages Oct 12 14:10:33 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 14:10:33 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 14:28:16 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 14:28:16 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 14:29:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e76d7a93-55b4-1e86-6751-3a9e278ca5ba (at 10.9.101.57@o2ib4) Oct 12 14:44:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Oct 12 14:49:44 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 14:49:44 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 14:55:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6647cce4-c951-cfd1-e471-985174791eba (at 10.9.112.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea9441000, cur 1539381352 expire 1539381202 last 1539381125 Oct 12 14:58:51 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 14:58:51 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 15:12:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 229af665-eaec-49e5-b8c2-2c97d6390c53 (at 10.9.112.8@o2ib4) Oct 12 15:19:21 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:22:54 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:22:54 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 15:23:35 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:23:35 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14 previous similar messages Oct 12 15:26:01 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:26:01 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 12 15:28:39 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:28:39 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 15:33:41 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:33:41 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Oct 12 15:41:33 oak-md1-s2 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 0s: evicting client at 10.9.105.65@o2ib4 ns: mdt-oak-MDT0000_UUID lock: ffff88171db93c00/0x6c00c4111a315f5b lrc: 4/0,0 mode: PR/PR res: [0x200017c29:0x858f:0x0].0x0 bits 0x20 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.105.65@o2ib4 remote: 0x42d0c1abff65eae7 expref: 6396 pid: 10517 timeout: 0 lvb_type: 0 Oct 12 15:45:53 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:45:53 oak-md1-s2 kernel: Lustre: 4310:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 28 previous similar messages Oct 12 15:48:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 12 15:49:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b4d7eebd-f59e-1a92-d3d6-5222338440b1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0e6800, cur 1539384566 expire 1539384416 last 1539384339 Oct 12 15:56:12 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 15:56:12 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 25 previous similar messages Oct 12 16:06:25 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 16:06:25 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 53 previous similar messages Oct 12 16:14:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d627c1e5-43d0-1b14-f5fc-7f0765ed8aa9 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881594e69c00, cur 1539386053 expire 1539385903 last 1539385826 Oct 12 16:17:15 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 16:17:15 oak-md1-s2 kernel: Lustre: 10655:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 38 previous similar messages Oct 12 16:28:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 12 16:28:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9335582-fbb1-527b-4898-0026893c6f76 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817f072e000, cur 1539386927 expire 1539386777 last 1539386700 Oct 12 16:32:15 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 16:32:15 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 85 previous similar messages Oct 12 16:32:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae9fe5e1-f384-5826-c900-bf63a2af2f8c (at 10.8.3.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ecceca400, cur 1539387163 expire 1539387013 last 1539386936 Oct 12 16:33:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8271ebc4-d739-add6-952f-6177412af161 (at 10.9.112.16@o2ib4) Oct 12 16:33:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2012ec85-a6fc-f7ce-e763-43d3a322b64e (at 10.8.3.14@o2ib6) Oct 12 16:58:10 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 16:58:10 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 16:59:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fba8e3f0-cb1c-da53-9886-24240e98ff92 (at 10.8.2.27@o2ib6) Oct 12 17:00:20 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 17:00:20 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 17:00:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 12 17:00:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 12 17:03:01 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 17:03:01 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 12 17:08:19 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 17:08:19 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 17:19:01 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 17:19:01 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 26 previous similar messages Oct 12 17:29:42 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 17:29:42 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 12 17:39:56 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 17:39:56 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages Oct 12 17:50:24 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 17:50:24 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 34 previous similar messages Oct 12 17:50:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aa20e722-f519-4016-3a01-1f0c34295a50 (at 10.9.107.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bee61d000, cur 1539391853 expire 1539391703 last 1539391626 Oct 12 17:54:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 12 17:54:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 12 18:00:33 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 18:00:33 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages Oct 12 18:14:22 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 18:14:22 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 12 18:25:36 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 18:25:36 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 12 18:30:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27bac3ac-d257-a563-4125-c911a485eb0b (at 10.9.101.60@o2ib4) Oct 12 18:40:02 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 18:40:02 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 12 18:54:26 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 18:54:26 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 19:08:24 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 19:08:24 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 19:18:50 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 19:18:50 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 19:34:15 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 19:34:15 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 19:45:39 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 19:45:39 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 19:55:59 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 19:55:59 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 20:06:47 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 20:06:47 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 12 20:11:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 156573be-eca5-3a48-3b0f-a12ef5bbd1ab (at 10.9.102.26@o2ib4) Oct 12 20:12:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe2e541b-a3dd-313c-a958-6ff6ca05d01e (at 10.9.102.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b95223400, cur 1539400351 expire 1539400201 last 1539400124 Oct 12 20:27:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e136347-39be-9972-2eef-9cc654f14efd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817d2694800, cur 1539401260 expire 1539401110 last 1539401033 Oct 12 20:27:43 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 20:27:43 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 12 20:33:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 12 20:34:12 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 20:34:12 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 20:36:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 12 20:37:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 307ba436-6256-ebe1-bc80-c60f08417038 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a9432bc00, cur 1539401849 expire 1539401699 last 1539401622 Oct 12 20:39:47 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 20:39:47 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 12 20:40:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b4cec00-47d5-f266-debf-715782b5a191 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b42aa7c00, cur 1539402017 expire 1539401867 last 1539401790 Oct 12 20:45:21 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 20:45:21 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 12 21:00:20 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 21:00:20 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14 previous similar messages Oct 12 21:11:46 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 21:11:46 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 12 22:00:25 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 22:00:25 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 12 22:02:19 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 22:12:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e05a18f2-2e4d-6c01-fc54-6f7f6a36a0aa (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe8698c00, cur 1539407579 expire 1539407429 last 1539407352 Oct 12 22:26:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 12 22:26:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ecde5eff-181a-c53f-f13d-f54852c52795 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88101b68e800, cur 1539408412 expire 1539408262 last 1539408185 Oct 12 22:32:11 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 22:32:11 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 12 22:32:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 12 22:33:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6158ed1c-a885-9a8b-673d-23f2334ff548 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881250752800, cur 1539408828 expire 1539408678 last 1539408601 Oct 12 22:35:14 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 22:37:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 12 22:38:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df19e90d-862b-5177-fbc6-ecba2dd1d13d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bf657d000, cur 1539409099 expire 1539408949 last 1539408872 Oct 12 22:41:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 12 22:41:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 31bc639a-fd40-2a3b-9273-41eb16c3a762 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881486021000, cur 1539409296 expire 1539409146 last 1539409069 Oct 12 22:41:36 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 22:45:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 12 22:46:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4de4ee1-54ac-33dc-4510-e6dad77506e7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a0da8fc00, cur 1539409573 expire 1539409423 last 1539409346 Oct 12 23:05:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42692ed8-a847-d651-433b-4cf3d056e150 (at 10.12.4.58@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9ac00, cur 1539410727 expire 1539410577 last 1539410500 Oct 12 23:15:55 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 12 23:28:12 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 00:01:48 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 00:07:03 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 00:25:08 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 00:37:55 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 00:58:29 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 00:58:29 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 13 01:08:57 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 01:08:57 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 01:58:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5dc470d-7939-6f03-58be-0fbd8766c4a7 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b3a3d000, cur 1539421136 expire 1539420986 last 1539420909 Oct 13 02:00:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 13 02:08:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d0f1863-893b-5fde-35f9-22e34a57eb82 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb84000, cur 1539421681 expire 1539421531 last 1539421454 Oct 13 02:08:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 13 02:15:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 13 02:16:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f31d222c-8a15-be39-ff76-6c9d2c03afab (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe774e800, cur 1539422182 expire 1539422032 last 1539421955 Oct 13 02:25:02 oak-md1-s2 kernel: Lustre: 4255:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 02:26:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a1dbfe0e-f658-7a93-f0a0-d9043a03d27b (at 10.12.4.73@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d39b4dc00, cur 1539422802 expire 1539422652 last 1539422575 Oct 13 02:27:31 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 02:48:10 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:00:12 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:00:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb1ee8e6-7ac7-9fe5-6ede-704dda9ea5d6 (at 10.12.4.30@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880353c3c800, cur 1539424857 expire 1539424707 last 1539424630 Oct 13 03:01:40 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:01:40 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 03:07:25 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:07:25 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 13 03:14:13 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:15:51 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:20:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 13 03:25:56 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:31:48 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:31:48 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 13 03:56:34 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:56:40 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:56:40 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 13 03:56:55 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 03:56:55 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 04:05:28 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:05:28 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 04:06:54 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:09:53 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:09:53 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 13 04:14:01 oak-md1-s2 kernel: Lustre: 10587:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:37:39 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:47:43 oak-md1-s2 kernel: Lustre: 10582:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:50:34 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:50:34 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 04:59:16 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 04:59:16 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 13 05:03:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:08:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:08:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe2e96a4-fe9b-2b74-5db6-6a651b98ab96 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802955b5800, cur 1539432534 expire 1539432384 last 1539432307 Oct 13 05:08:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 13 05:12:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:12:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9f4ace39-9733-acd7-4141-be6c695a28dd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005b2bc000, cur 1539432747 expire 1539432597 last 1539432520 Oct 13 05:15:20 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 05:15:20 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 05:16:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:16:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 40bcccda-9ffd-9b5b-82d3-fdda8216d513 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ed256800, cur 1539433004 expire 1539432854 last 1539432777 Oct 13 05:20:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:21:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 60999113-58a3-b7f4-37b2-9006c4fba15a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c43be3800, cur 1539433286 expire 1539433136 last 1539433059 Oct 13 05:24:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:24:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fed14e1d-bb55-7eda-e5ba-cd55d7a35e40 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb73dc000, cur 1539433497 expire 1539433347 last 1539433270 Oct 13 05:28:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:28:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a907b408-8e93-a903-abce-d0d2075fb0d1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c5f6a8400, cur 1539433727 expire 1539433577 last 1539433500 Oct 13 05:34:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:34:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 827716c4-e0da-47e0-9b2a-fbf39ab30108 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184f709000, cur 1539434089 expire 1539433939 last 1539433862 Oct 13 05:36:29 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 05:39:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 05:40:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 00e4d9c4-d7a8-2bc4-7095-3749157e65de (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e4284a000, cur 1539434424 expire 1539434274 last 1539434197 Oct 13 05:47:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 202e657e-05ef-cb2a-6989-0480102735cf (at 10.8.6.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880877ceb400, cur 1539434836 expire 1539434686 last 1539434609 Oct 13 05:48:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4e89e1fc-b9e8-2d83-fe2a-4e36f66d881f (at 10.8.6.4@o2ib6) Oct 13 06:00:48 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 06:00:48 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 06:13:40 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 06:23:37 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 06:25:46 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 06:25:46 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 07:03:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4182f089-3d11-842f-80e8-d4f66c26a376 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef00bc00, cur 1539439409 expire 1539439259 last 1539439182 Oct 13 07:03:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 13 07:11:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a5041eaf-fd8e-8d42-63ef-016835994615 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe774d400, cur 1539439904 expire 1539439754 last 1539439677 Oct 13 07:12:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 13 07:26:11 oak-md1-s2 kernel: Lustre: 10572:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 07:30:46 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 07:33:35 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 07:34:38 oak-md1-s2 kernel: Lustre: 10555:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 07:54:43 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 08:03:09 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 08:03:09 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 08:32:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c39a9723-c48a-d4e2-9fc0-d3c430d4be39 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fceac2c00, cur 1539444760 expire 1539444610 last 1539444533 Oct 13 08:33:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 13 08:51:43 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 08:51:43 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 13 09:15:34 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 09:19:50 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 09:29:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4be195d8-2ce9-9f63-3d76-19186e9fe953 (at 10.12.4.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880063422000, cur 1539448174 expire 1539448024 last 1539447947 Oct 13 09:30:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 22e86b33-5ed6-4f3c-7486-e2e7ff656408 (at 10.12.4.25@o2ib) Oct 13 09:30:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f97f5c4e-ea1b-a56f-2caf-9e8a029f9f0e (at 10.12.4.30@o2ib) Oct 13 09:30:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 13 09:35:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0c7eefa3-fc99-540c-06e7-6b75b4a7596a (at 10.12.4.71@o2ib) Oct 13 09:38:06 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:04:10 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:11:00 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:11:00 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 13 10:15:59 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:16:43 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:16:43 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 10:24:18 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:24:30 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:28:58 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:33:10 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 10:58:01 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 12:10:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cd6ba70e-5bc5-df20-1777-946e4c58b925 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e71c8400, cur 1539457811 expire 1539457661 last 1539457584 Oct 13 12:29:15 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 13:15:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 947b9ee4-21df-eb8e-f730-826f785aab58 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c64a84c00, cur 1539461739 expire 1539461589 last 1539461512 Oct 13 13:21:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 557acccb-6452-a62c-9771-8fffc8bf5f70 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185c29c000, cur 1539462118 expire 1539461968 last 1539461891 Oct 13 13:22:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 13:26:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 13:27:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea2427a2-6f4e-82d8-10f1-51d5987b6680 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881596263400, cur 1539462432 expire 1539462282 last 1539462205 Oct 13 14:34:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 986bf173-8a48-8326-0b87-c02bc7db3b53 (at 10.8.9.8@o2ib6) Oct 13 14:35:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7bdfc76-9bed-36bb-1a90-fd5366821923 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dc196c800, cur 1539466512 expire 1539466362 last 1539466285 Oct 13 15:33:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 751a42b3-a5b4-b9d3-9530-93af4d5fb366 (at 10.8.6.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e151e8400, cur 1539470016 expire 1539469866 last 1539469789 Oct 13 15:34:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Oct 13 15:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 642a05d1-cec4-e1d4-5e6c-b598c09798dd (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031f761400, cur 1539470589 expire 1539470439 last 1539470362 Oct 13 17:01:56 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 17:58:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2413480f-ea05-5d69-bbd1-9a47d12c9895 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818987d9800, cur 1539478737 expire 1539478587 last 1539478510 Oct 13 18:11:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65662b8f-8950-468b-c07a-eed17ba40c1f (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a978f8800, cur 1539479515 expire 1539479365 last 1539479288 Oct 13 18:12:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 13 18:14:36 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 18:33:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2393845d-6c59-3627-70e3-8509fc556ddc (at 10.12.4.38@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e6624000, cur 1539480818 expire 1539480668 last 1539480591 Oct 13 18:34:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 724421ba-8d38-1f4f-76df-cf665bbb36f7 (at 10.8.18.35@o2ib6) in 155 seconds. I think it's dead, and I am evicting it. exp ffff881e25d21000, cur 1539480894 expire 1539480744 last 1539480739 Oct 13 18:35:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 18:35:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to edf32d0b-88eb-f1d4-00b8-c0af2b7893e4 (at 10.12.4.38@o2ib) Oct 13 18:43:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 18:43:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 853050c7-f06c-a2fc-3bf1-af24cb12ce9b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe69dcc00, cur 1539481422 expire 1539481272 last 1539481195 Oct 13 18:48:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 18:49:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe8b0512-9da7-7d20-e657-286a58dff07f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053efc400, cur 1539481769 expire 1539481619 last 1539481542 Oct 13 18:54:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 18:54:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8c170c2b-edbb-a2b2-56c6-296d3c8b8c43 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b68e2800, cur 1539482063 expire 1539481913 last 1539481836 Oct 13 18:54:29 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 18:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 18:59:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 968d9a69-cc34-95e8-14e0-dd33c783e95c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fc48cbc00, cur 1539482372 expire 1539482222 last 1539482145 Oct 13 19:03:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 19:03:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b33e778-1304-d169-7a56-455df6b55c2a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f3ccfcc00, cur 1539482639 expire 1539482489 last 1539482412 Oct 13 19:06:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 19:07:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9d313167-d44e-b189-9778-f1c44098c064 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b99264000, cur 1539482860 expire 1539482710 last 1539482633 Oct 13 19:12:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4686b3f8-16e5-819e-63be-2143450a99c3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001d9e000, cur 1539483141 expire 1539482991 last 1539482914 Oct 13 19:14:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 19:26:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 19:26:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5c05e9f2-c98d-2e7f-1187-1ff5670f8208 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210ac00, cur 1539484000 expire 1539483850 last 1539483773 Oct 13 19:30:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 71a7c9e2-526c-21b9-1808-45b9b3c143c7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881befb81c00, cur 1539484205 expire 1539484055 last 1539483978 Oct 13 19:35:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dfc0bca0-cf14-b6c8-69fe-692fc54d4e60 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88059c29ec00, cur 1539484556 expire 1539484406 last 1539484329 Oct 13 19:36:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 19:36:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 13 19:41:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2bc7382-ff61-f696-4e9c-8367d3cffef4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880038e88c00, cur 1539484914 expire 1539484764 last 1539484687 Oct 13 19:52:11 oak-md1-s2 kernel: Lustre: 3867:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 19:55:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 19:55:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 13 19:55:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43f4759d-9652-fc3a-9f90-f25068649dce (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881246ae3400, cur 1539485725 expire 1539485575 last 1539485498 Oct 13 19:58:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f6ee4a49-d30b-32bb-dcd5-73a2d99619a7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882009850400, cur 1539485928 expire 1539485778 last 1539485701 Oct 13 20:02:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1640bcc7-c77d-0b57-cfc7-9a21d2d13150 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dc196c000, cur 1539486169 expire 1539486019 last 1539485942 Oct 13 20:03:01 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 20:03:01 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 20:06:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fba9b3c3-6ead-e6ba-8c87-c0aca545e9a3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804e174a000, cur 1539486402 expire 1539486252 last 1539486175 Oct 13 20:08:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 20:08:46 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 13 20:12:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 897372aa-91fc-6636-ebd7-f1951e5d3756 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880026c79400, cur 1539486779 expire 1539486629 last 1539486552 Oct 13 20:21:12 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 20:21:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 20:21:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 13 20:21:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1b1cc713-4c0b-77df-c66a-6df48467f5b2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880877ce8400, cur 1539487315 expire 1539487165 last 1539487088 Oct 13 20:21:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 13 20:37:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 20:37:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 13 20:38:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cd5118d9-8e29-e448-15c2-3d5c2b824abb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1c0bcc00, cur 1539488307 expire 1539488157 last 1539488080 Oct 13 20:38:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 13 20:49:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3fe607ed-106b-2d21-cf35-b1b71ed5ee1c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184d7af000, cur 1539488960 expire 1539488810 last 1539488733 Oct 13 20:49:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 13 20:49:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 13 20:49:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 13 20:57:14 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:09:33 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:11:35 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:11:35 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 21:16:24 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:21:53 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:23:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 88f0832a-08ab-8808-ec4c-f9fd62681672 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881898684400, cur 1539491011 expire 1539490861 last 1539490784 Oct 13 21:26:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 13 21:28:02 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:30:23 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:32:09 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:32:09 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 21:32:43 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:32:43 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 13 21:37:36 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 21:37:36 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 22:38:59 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 22:38:59 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 13 22:45:31 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 13 22:45:31 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 13 23:29:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 13 23:30:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4d3a0d78-72aa-308b-1756-d30e46e4a590 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818292afc00, cur 1539498643 expire 1539498493 last 1539498416 Oct 14 00:09:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 00:10:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2ad3e3d6-277e-4911-13fe-f0a3e9863b98 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802955b5800, cur 1539501042 expire 1539500892 last 1539500815 Oct 14 00:37:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 00:38:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bbf75cf-685f-5d5d-085e-2064dd4b973b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e8244c00, cur 1539502721 expire 1539502571 last 1539502494 Oct 14 00:46:59 oak-md1-s2 kernel: Lustre: 10588:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 00:48:14 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 00:48:14 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 01:09:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 01:09:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9cced365-8736-9d96-c08f-3c5ea710c0c7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881befb81c00, cur 1539504598 expire 1539504448 last 1539504371 Oct 14 01:41:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 01:42:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c4dcd044-6216-8503-616e-ba2f20163efb (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800500c0000, cur 1539506537 expire 1539506387 last 1539506310 Oct 14 01:53:00 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 01:56:07 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 02:07:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:07:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c36b8e04-5ccb-2a0e-6b7d-8ee4a0291a3a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031f760400, cur 1539508074 expire 1539507924 last 1539507847 Oct 14 02:15:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:16:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9cf3751-feb8-ebfb-f76c-8fcbbce19e8c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c9eb5bc00, cur 1539508571 expire 1539508421 last 1539508344 Oct 14 02:20:31 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 02:23:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5b1c15b3-774c-93c5-d10b-d7d3cf9e8761 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019e612c00, cur 1539509000 expire 1539508850 last 1539508773 Oct 14 02:23:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:26:52 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 02:26:52 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 02:28:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:29:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd2bffaf-1409-5fdb-0111-c4b4089d0240 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba5000, cur 1539509341 expire 1539509191 last 1539509114 Oct 14 02:29:21 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 02:33:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 048e6b2d-bcd7-287a-3cc1-c03126bbb4c6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db00d5c00, cur 1539509589 expire 1539509439 last 1539509362 Oct 14 02:33:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:38:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 24619fe5-8217-fa93-2583-fdfd59e99bd2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d7cbc00, cur 1539509897 expire 1539509747 last 1539509670 Oct 14 02:38:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:44:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7fdb6eb8-8988-4af6-64d0-091e0b92b7b8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4b592c00, cur 1539510253 expire 1539510103 last 1539510026 Oct 14 02:44:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:45:35 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 02:50:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 02:50:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 743538f6-603d-ece7-9208-83fbde67a025 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096a0f4c00, cur 1539510605 expire 1539510455 last 1539510378 Oct 14 02:55:06 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 02:55:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 63cc4494-146a-2737-a5f5-6426cb5ab1d0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88128377f000, cur 1539510954 expire 1539510804 last 1539510727 Oct 14 02:56:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 03:00:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 03:00:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 62670b5a-51fd-991b-a461-852dfdb43f45 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88148eef1800, cur 1539511225 expire 1539511075 last 1539510998 Oct 14 03:01:56 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:01:56 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 03:05:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 03:05:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f0b5932-70d2-86ce-de5e-0ca9511318f9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2117000, cur 1539511511 expire 1539511361 last 1539511284 Oct 14 03:08:32 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:08:48 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:08:48 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 03:13:15 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:14:06 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:15:27 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:18:14 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:23:31 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:32:36 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:32:36 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 03:41:33 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:41:33 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14 previous similar messages Oct 14 03:50:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 03:51:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0bd01329-08cf-1cda-80ec-76ab717f484a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880075af7400, cur 1539514302 expire 1539514152 last 1539514075 Oct 14 03:53:07 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 03:53:07 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 14 03:59:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 03:59:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 899865ea-59f7-27d3-99db-738fd8ec48b8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d0e236800, cur 1539514786 expire 1539514636 last 1539514559 Oct 14 04:03:08 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 04:03:08 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 14 04:03:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90d31ab8-8741-2e29-9802-a03d8777d511 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096a0f4c00, cur 1539515026 expire 1539514876 last 1539514799 Oct 14 04:03:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 04:08:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 218bc340-a1e0-2cfe-d2d6-c860bf462f09 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d0e233800, cur 1539515290 expire 1539515140 last 1539515063 Oct 14 04:08:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 04:08:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 04:16:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 80e8ddaa-aa8c-cfa3-6112-24516d8544b4 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881473643000, cur 1539515804 expire 1539515654 last 1539515577 Oct 14 04:16:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 04:21:09 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 04:21:09 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 14 04:25:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 04:25:34 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 04:34:00 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 04:34:00 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 14 04:37:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e30a512-a239-5c55-45aa-eff317c52230 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88076f78d800, cur 1539517062 expire 1539516912 last 1539516835 Oct 14 04:37:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 04:41:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 04:41:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 04:44:52 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 04:44:52 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 05:03:21 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 05:03:21 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 05:05:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 05:06:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fabdef4d-45dd-ee97-aad1-82e9d287ce11 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88050764ac00, cur 1539518794 expire 1539518644 last 1539518567 Oct 14 05:13:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 05:13:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e83ef8f4-43cb-e7f6-5cca-a56c83a7cbcb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f0c00, cur 1539519238 expire 1539519088 last 1539519011 Oct 14 05:18:06 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 05:22:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 05:22:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7a527019-1cbc-8dee-9216-ea2c7b8cb57e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a71f000, cur 1539519757 expire 1539519607 last 1539519530 Oct 14 05:30:32 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 05:30:32 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 05:40:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 05:41:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 230a6b3c-7fa1-be76-fb85-a4fb1f931df5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880af5e3b800, cur 1539520873 expire 1539520723 last 1539520646 Oct 14 05:44:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 05:45:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b06512e-2745-3b3d-ffae-efc23f0ec36d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb73db000, cur 1539521108 expire 1539520958 last 1539520881 Oct 14 05:46:20 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 05:46:20 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 14 05:49:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 05:49:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c40d623-edbb-674c-6efa-5026950a456c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e7e7ac00, cur 1539521395 expire 1539521245 last 1539521168 Oct 14 05:53:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 05:54:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98fd577a-6b9e-0f69-c014-a96edd45fca0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b7ae4c800, cur 1539521653 expire 1539521503 last 1539521426 Oct 14 05:57:09 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 05:57:09 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 06:05:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51d590bd-d8d9-6d26-fdd2-40b78596c75c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc0808400, cur 1539522326 expire 1539522176 last 1539522099 Oct 14 06:06:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 06:10:21 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 06:11:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 19274ac8-5fdf-33b2-57ae-bdd0a4377601 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880810d0a800, cur 1539522670 expire 1539522520 last 1539522443 Oct 14 06:16:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 16011f98-1441-5d20-e5b8-9fcd485a3c16 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f5800, cur 1539522974 expire 1539522824 last 1539522747 Oct 14 06:20:35 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 06:21:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 77d0d051-69a2-b771-cda6-f8523ad60348 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818692d9000, cur 1539523278 expire 1539523128 last 1539523051 Oct 14 06:21:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 06:21:29 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 06:29:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a50fd6b-c5b2-0d94-a43e-928053cc9e2b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef094800, cur 1539523792 expire 1539523642 last 1539523565 Oct 14 06:34:30 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 06:34:30 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 06:35:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 039d0394-8801-5c40-f355-0b0c5e808fa9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1f6ad400, cur 1539524146 expire 1539523996 last 1539523919 Oct 14 06:36:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 06:36:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 06:40:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f503f0af-8e39-e1f4-3ac8-061041d4c60d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f96a56400, cur 1539524427 expire 1539524277 last 1539524200 Oct 14 06:45:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8cbdf304-c411-498a-cc29-5cfc55d82d14 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b68e0800, cur 1539524735 expire 1539524585 last 1539524508 Oct 14 06:48:12 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 06:48:12 oak-md1-s2 kernel: Lustre: 4303:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 06:55:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 06:55:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 06:55:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4eb821c0-2174-fa5a-f7da-522f0b350255 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db0385000, cur 1539525358 expire 1539525208 last 1539525131 Oct 14 06:58:49 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 07:08:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3147b77a-84d8-2750-1c8f-c5c6ef960766 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040e0b800, cur 1539526083 expire 1539525933 last 1539525856 Oct 14 07:08:03 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 07:08:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 07:08:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 07:09:06 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 07:09:06 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 07:21:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f84d9e02-a65e-cf00-9c5f-1ecab6244024 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a5f779800, cur 1539526910 expire 1539526760 last 1539526683 Oct 14 07:21:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 07:21:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 07:21:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 07:32:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5457b59-608a-a1ad-0511-692bcb6b33fb (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68d000, cur 1539527555 expire 1539527405 last 1539527328 Oct 14 07:32:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 07:33:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 07:33:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 07:38:23 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 07:38:23 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 07:47:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 07:47:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 07:47:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e43bd252-80cd-5650-0237-f8312aeea220 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb8d800, cur 1539528453 expire 1539528303 last 1539528226 Oct 14 07:47:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 07:59:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 07:59:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 08:00:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fcedd23b-2749-c3c7-1046-48ab3a0af158 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001dd4c00, cur 1539529204 expire 1539529054 last 1539528977 Oct 14 08:00:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 08:17:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 08:17:35 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 08:17:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cc7a4254-a4d6-8b65-0c81-6306f782d933 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa96e800, cur 1539530256 expire 1539530106 last 1539530029 Oct 14 08:17:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 08:21:55 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:21:55 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 14 08:24:17 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:25:10 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:25:10 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 08:27:11 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:27:11 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 08:31:22 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:31:22 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 08:35:47 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:35:47 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 08:37:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 08:37:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 08:37:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b733dcd-8da2-6967-79e6-f1f87a3c9040 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afa800, cur 1539531468 expire 1539531318 last 1539531241 Oct 14 08:37:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 08:41:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e8a2a567-1568-a659-f448-49ec1e8210ce (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103c097400, cur 1539531691 expire 1539531541 last 1539531464 Oct 14 08:42:50 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:42:50 oak-md1-s2 kernel: Lustre: 10579:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 08:47:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1d79ccaf-59b2-f5ff-0e01-110c72bdd363 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf93c000, cur 1539532020 expire 1539531870 last 1539531793 Oct 14 08:51:47 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539532300/real 1539532300] req@ffff8800164f2d00 x1610632628552432/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539532307 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 14 08:51:47 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 14 08:52:01 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539532314/real 1539532314] req@ffff8800164f2d00 x1610632628552432/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539532321 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 14 08:52:01 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 14 08:52:22 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539532335/real 1539532335] req@ffff8800164f2d00 x1610632628552432/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539532342 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 14 08:52:22 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 14 08:53:04 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539532377/real 1539532377] req@ffff8800164f2d00 x1610632628552432/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539532384 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 14 08:53:04 oak-md1-s2 kernel: Lustre: 10574:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 14 08:53:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 08:53:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 08:53:12 oak-md1-s2 kernel: LustreError: 10574:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.18.35@o2ib6) returned error from blocking AST (req@ffff8800164f2d00 x1610632628552432 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff88006080b200/0x6c00c417b1a17080 lrc: 4/0,0 mode: PR/PR res: [0x200013483:0x19b76:0x0].0x0 bits 0x13 rrc: 8 type: IBT flags: 0x60200400000020 nid: 10.8.18.35@o2ib6 remote: 0xf3d1693c77152c52 expref: 26 pid: 10526 timeout: 7816734937 lvb_type: 0 Oct 14 08:53:12 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.18.35@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 14 08:53:50 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 08:53:50 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 14 08:58:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98b9e4d6-b6a7-827a-2a1e-13ac8b540d89 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1f000, cur 1539532737 expire 1539532587 last 1539532510 Oct 14 09:03:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 14 09:03:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 09:08:56 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 09:08:56 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages Oct 14 09:19:50 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 09:19:50 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages Oct 14 09:22:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7597169f-f487-3bba-1dde-55515364af21 (at 10.8.8.14@o2ib6) reconnecting Oct 14 09:22:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 556bf39b-de29-e355-586b-a885439bd78c (at 10.8.8.14@o2ib6) Oct 14 09:31:17 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 09:31:17 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages Oct 14 09:42:28 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 09:42:28 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 14 previous similar messages Oct 14 09:51:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a03eaf56-bde2-a306-7dcc-8d03b12dba34 (at 10.12.4.59@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e19c42000, cur 1539535899 expire 1539535749 last 1539535672 Oct 14 09:51:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 09:57:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4545744-f692-2d85-b8b7-c85b71f74d7b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200ea66000, cur 1539536247 expire 1539536097 last 1539536020 Oct 14 09:57:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 09:57:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 09:57:58 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 09:57:58 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 14 10:02:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 10:02:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 50ab1cb5-67ef-cba6-fca1-2f72e17993ed (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881beeebc000, cur 1539536535 expire 1539536385 last 1539536308 Oct 14 10:05:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 589340ed-28cc-9e4a-ea13-515b211cd26f (at 10.0.2.230@o2ib5) Oct 14 10:10:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8832746f-076c-9f89-5338-dfbfb3bd306e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb93c00, cur 1539537046 expire 1539536896 last 1539536819 Oct 14 10:10:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 10:10:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 10:10:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 10:12:52 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 10:12:52 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 14 10:22:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 684efc72-d56c-9a4c-08c6-104a9c7edcb4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1f000, cur 1539537777 expire 1539537627 last 1539537550 Oct 14 10:22:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 10:23:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 10:23:06 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 10:24:01 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 10:24:01 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 10:34:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 10:34:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 10:35:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client af8c506b-c113-32fe-4706-f8f3056ffd94 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804bc21c400, cur 1539538527 expire 1539538377 last 1539538300 Oct 14 10:35:27 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 10:36:30 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 10:36:30 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 10:46:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 10:46:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 10:47:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf2f16ce-a261-ab6e-6ded-c0e18a52deff (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc71d9000, cur 1539539248 expire 1539539098 last 1539539021 Oct 14 10:47:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 10:50:17 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 10:50:17 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 14 10:56:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 10:56:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 10:58:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 56374a8c-1030-d55b-034c-d98af1067209 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018dd73c00, cur 1539539911 expire 1539539761 last 1539539684 Oct 14 10:58:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 11:02:20 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 11:02:20 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 11:09:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 11:09:25 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 11:09:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b76679eb-6a6e-12eb-8d1f-5ea1949e4b0f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88179c231400, cur 1539540588 expire 1539540438 last 1539540361 Oct 14 11:09:48 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 11:12:29 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 11:12:29 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 11:19:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 11:19:37 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 11:19:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 570483a6-45b5-dcc6-c2af-a6bc293fb954 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b00684c00, cur 1539541193 expire 1539541043 last 1539540966 Oct 14 11:19:53 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 11:29:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 11:29:59 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 11:30:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3ae63f60-ae9b-bfdb-336f-8ee2e8468e62 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e4ac23000, cur 1539541830 expire 1539541680 last 1539541603 Oct 14 11:30:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 11:33:16 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 11:33:16 oak-md1-s2 kernel: Lustre: 10674:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 14 11:38:38 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 11:40:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 11:40:49 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 11:41:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce48ed02-506f-1b9a-86d8-4d53d257c692 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daccb7000, cur 1539542491 expire 1539542341 last 1539542264 Oct 14 11:41:31 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 11:53:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 11:53:16 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 11:53:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f384614-cbd9-f7c5-d4f6-40ee9c89ace6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e71cf400, cur 1539543218 expire 1539543068 last 1539542991 Oct 14 11:53:38 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 12:03:08 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:03:08 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 12:05:55 oak-md1-s2 kernel: Lustre: 5518:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:06:28 oak-md1-s2 kernel: Lustre: DEBUG MARKER: Sun Oct 14 12:06:28 2018 Oct 14 12:07:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e50fcb87-93cc-a71e-c4e3-bc5b96a49e73 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801530ed000, cur 1539544039 expire 1539543889 last 1539543812 Oct 14 12:07:19 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 12:08:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 12:08:10 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 12:12:55 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:15:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7980f9e0-a876-9d0e-aaa3-7ea63f72dc7a (at 10.0.2.228@o2ib5) reconnecting Oct 14 12:15:38 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:18:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 12:18:26 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 12:19:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cdbef1cf-4070-72b9-6dc3-e0382d153f29 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805adfaa000, cur 1539544760 expire 1539544610 last 1539544533 Oct 14 12:19:20 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 12:20:25 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:20:25 oak-md1-s2 kernel: Lustre: 10636:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 12:30:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 12:30:07 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 12:30:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef8220d4-5541-563c-9efb-0059dfff0499 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880353c3a400, cur 1539545409 expire 1539545259 last 1539545182 Oct 14 12:30:09 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 12:36:10 oak-md1-s2 kernel: Lustre: 10563:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:39:53 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:39:53 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 12:42:29 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:42:29 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 12:46:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 12:46:00 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 12:46:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d148c396-d428-ed8a-4c01-7d4de3dcf96c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d6e211400, cur 1539546376 expire 1539546226 last 1539546149 Oct 14 12:46:16 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 12:47:05 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 12:58:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 12:58:35 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 12:59:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dee7685d-8868-e050-fa47-dfb6ebd5ce20 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880150eff000, cur 1539547157 expire 1539547007 last 1539546930 Oct 14 12:59:17 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 13:06:08 oak-md1-s2 kernel: Lustre: 10507:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 13:08:40 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 13:08:40 oak-md1-s2 kernel: Lustre: 10589:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 13:10:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9a48a43f-da54-c4ed-9e4a-85e206c95369 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815e56fec00, cur 1539547810 expire 1539547660 last 1539547583 Oct 14 13:10:10 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:10:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 13:10:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:13:43 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 13:13:43 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 13:19:08 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 13:19:08 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 14 13:20:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a0adfd5d-01e6-94c4-f13f-b1f1cab1e837 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa226000, cur 1539548456 expire 1539548306 last 1539548229 Oct 14 13:20:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:23:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 13:23:19 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 13:24:43 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 13:31:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c1fd8989-a28c-e77f-5ca2-6f879efddf23 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a53c00, cur 1539549082 expire 1539548932 last 1539548855 Oct 14 13:31:22 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:34:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 13:34:42 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:38:47 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 13:38:47 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 14 13:42:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74a7765d-a404-3380-2f12-588161506789 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88128aa05c00, cur 1539549727 expire 1539549577 last 1539549500 Oct 14 13:42:07 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:45:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 13:45:02 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:51:11 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 13:51:11 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 14 13:57:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f5ef654-c978-b63d-4594-10bc5df2e5f5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf6f07800, cur 1539550645 expire 1539550495 last 1539550418 Oct 14 13:57:25 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 13:57:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 13:57:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 14:07:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cd6fd6e2-35f1-e55a-54ab-6fabf6de921c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8820098dd400, cur 1539551254 expire 1539551104 last 1539551027 Oct 14 14:07:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 14:07:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 14:07:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 14:14:18 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 14:14:18 oak-md1-s2 kernel: Lustre: 10526:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 14:17:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90dcb950-cb0b-91cd-3849-a33e36084263 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b35317800, cur 1539551856 expire 1539551706 last 1539551629 Oct 14 14:17:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 14:19:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 14:19:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 14:24:39 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 14:24:39 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 14:25:40 oak-md1-s2 kernel: Lustre: 10596:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539552333/real 1539552333] req@ffff881a46b4f800 x1610632659370400/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539552340 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 14 14:25:40 oak-md1-s2 kernel: Lustre: 10596:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 14 14:25:48 oak-md1-s2 kernel: LustreError: 10596:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.17.26@o2ib6) returned error from blocking AST (req@ffff881a46b4f800 x1610632659370400 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880025dc0000/0x6c00c417ee24cd16 lrc: 4/0,0 mode: PR/PR res: [0x200013483:0x19b76:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.17.26@o2ib6 remote: 0x6443cb0c11634e12 expref: 21 pid: 3871 timeout: 7836690963 lvb_type: 0 Oct 14 14:25:48 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.17.26@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 14 14:28:08 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 14:28:08 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 14:29:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e34d494-8aa6-75b0-9ca2-64727d879b30 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881596262c00, cur 1539552573 expire 1539552423 last 1539552346 Oct 14 14:29:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 14:33:31 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 14:33:31 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 14:33:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 14:33:44 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 14:41:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51f8c5f9-85ee-c335-c20d-2e30dbde109c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea36f8c00, cur 1539553283 expire 1539553133 last 1539553056 Oct 14 14:41:23 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 14:48:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 14:48:11 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 14:50:01 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 14:50:01 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 14 14:52:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 837482ac-2d00-0262-db98-1cbfd90f0eaf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf27d400, cur 1539553964 expire 1539553814 last 1539553737 Oct 14 14:52:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 15:00:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 15:00:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 15:02:04 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 15:02:04 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 15:14:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 14 15:14:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7dbcbfe4-df9d-7bee-8fb8-1770f1930819 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a7000, cur 1539555272 expire 1539555122 last 1539555045 Oct 14 15:14:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 15:20:31 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 15:33:48 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 15:33:48 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 15:35:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 15:35:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 08b206ba-57cb-e66c-7fdf-e3873e08db4d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7594400, cur 1539556505 expire 1539556355 last 1539556278 Oct 14 15:39:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 14 15:40:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 72f012cc-1405-1296-39e8-391ef6aa1e96 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b3a3bc00, cur 1539556852 expire 1539556702 last 1539556625 Oct 14 15:44:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 15:45:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33240ee0-8209-1aff-db97-8232fd457370 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096a0f7400, cur 1539557153 expire 1539557003 last 1539556926 Oct 14 15:46:52 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 15:46:52 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 15:49:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 14 15:50:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90085f90-51a4-00bf-1ab8-d38d18ad906a (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039b5a6000, cur 1539557438 expire 1539557288 last 1539557211 Oct 14 15:54:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb8d4145-f147-10be-82db-2d3c08a09513 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bdeb8b400, cur 1539557647 expire 1539557497 last 1539557420 Oct 14 16:02:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 16:02:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 16:03:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e725d92-04f3-0ce1-6c7d-b48e3f675a85 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880046e2c800, cur 1539558182 expire 1539558032 last 1539557955 Oct 14 16:12:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9d2d4d6-d4e6-cd06-1f22-52b678f2c237 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bef734c00, cur 1539558738 expire 1539558588 last 1539558511 Oct 14 16:12:18 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 16:16:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 14 16:16:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 16:17:29 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 16:17:29 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 16:23:11 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 16:26:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 16:26:15 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 16:26:15 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 14 16:26:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3d40453d-c683-6341-4475-021cdfbd260a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018dd71800, cur 1539559601 expire 1539559451 last 1539559374 Oct 14 16:26:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 16:43:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 16:43:35 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 16:44:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db51e95c-ee03-fc06-5103-6e9d1da317ad (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804eee23400, cur 1539560656 expire 1539560506 last 1539560429 Oct 14 16:44:16 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 16:56:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 16:56:46 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 16:57:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7e8ef184-1daa-4728-85cb-1260e0b4dd4a (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc71dd400, cur 1539561473 expire 1539561323 last 1539561246 Oct 14 16:57:53 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 17:03:31 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 17:03:31 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 17:07:29 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 17:07:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 17:07:57 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 17:10:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df0413ca-bd69-1453-5aca-f7f6cc845dab (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab0863400, cur 1539562212 expire 1539562062 last 1539561985 Oct 14 17:10:12 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 17:13:08 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 17:13:08 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 17:20:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 17:20:13 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 17:20:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6bd32066-27d0-f52e-a174-598c77caeb9b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e783400, cur 1539562836 expire 1539562686 last 1539562609 Oct 14 17:20:36 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 17:32:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 17:32:24 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 14 17:33:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ea1a802-440e-2d14-fca4-01431917eead (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881849fd1000, cur 1539563595 expire 1539563445 last 1539563368 Oct 14 17:33:15 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 14 17:37:07 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 17:37:07 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 17:43:18 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 17:44:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 17:44:17 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 14 17:45:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f9d6cb5-89ee-0b3b-4de9-c4c145d4edbc (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005b9a7400, cur 1539564325 expire 1539564175 last 1539564098 Oct 14 17:45:25 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 14 18:00:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 18:00:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 18:00:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c2938cd-e17a-7b98-9d83-52d8c7da66e5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b66a27400, cur 1539565226 expire 1539565076 last 1539564999 Oct 14 18:00:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 18:10:16 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 18:14:29 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 18:14:29 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 18:27:09 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 18:56:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 84f451cf-2084-4ecc-faae-166857bdc98d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808552a6c00, cur 1539568598 expire 1539568448 last 1539568371 Oct 14 18:56:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:01:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a005fe82-7363-0a7b-8cfd-222b993f96aa (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc71ddc00, cur 1539568919 expire 1539568769 last 1539568692 Oct 14 19:02:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:05:37 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 19:05:37 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 19:05:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:05:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 19:06:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 836ec550-dd67-de4f-ceca-45d4ec580ca8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ce878cc00, cur 1539569191 expire 1539569041 last 1539568964 Oct 14 19:06:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 19:11:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:11:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 19:16:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2bc281c-c628-146e-5d46-1c593c075e64 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf36b3000, cur 1539569783 expire 1539569633 last 1539569556 Oct 14 19:16:23 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 19:23:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:23:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 19:32:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94cd4e3f-4941-fac6-b9e8-6dc005da6ba5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef00fc00, cur 1539570765 expire 1539570615 last 1539570538 Oct 14 19:32:45 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 14 19:37:47 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 19:37:47 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 19:38:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:38:59 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 19:47:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f261731c-f06f-07c9-c74c-60d134d7d359 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227c9000, cur 1539571642 expire 1539571492 last 1539571415 Oct 14 19:47:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 19:47:47 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 19:49:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:49:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 19:59:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 19:59:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 20:00:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 872a3037-d6b3-25d6-64f2-514fe17d17a7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881875ecc400, cur 1539572424 expire 1539572274 last 1539572197 Oct 14 20:00:24 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 20:10:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 20:10:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 20:11:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 03613dea-e5c1-da7f-2817-6dc075cc27e8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881691145000, cur 1539573085 expire 1539572935 last 1539572858 Oct 14 20:11:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 20:23:00 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:23:00 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 20:23:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 20:23:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 20:24:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa1c80cf-59f8-6df4-16b6-899a521ceab5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dab764000, cur 1539573892 expire 1539573742 last 1539573665 Oct 14 20:24:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 20:27:53 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:31:29 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:31:54 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:35:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 20:35:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 20:36:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39c397f2-2c23-8fec-5cf8-491c0629fcb1 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db1c6b000, cur 1539574600 expire 1539574450 last 1539574373 Oct 14 20:36:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 20:45:04 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:48:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 20:48:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 20:48:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a3cd107-3b7d-ce49-fa7e-c3cb21da888d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005dcb0000, cur 1539575339 expire 1539575189 last 1539575112 Oct 14 20:48:59 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 20:52:40 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:58:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 20:58:33 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 20:58:33 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:59:19 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 20:59:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 58ac5d0b-2819-974b-035a-d8fe8fc61e02 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005771e400, cur 1539575964 expire 1539575814 last 1539575737 Oct 14 20:59:24 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 21:02:29 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 21:02:38 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 21:02:38 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 21:13:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 21:13:21 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 21:14:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ff6a146-0177-3982-b3c9-7d12f373f6e5 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c29685c00, cur 1539576853 expire 1539576703 last 1539576626 Oct 14 21:14:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 21:15:50 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 21:15:50 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 21:17:33 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 21:17:33 oak-md1-s2 kernel: Lustre: 10632:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 21:20:07 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 21:21:19 oak-md1-s2 kernel: Lustre: 3868:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 21:23:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 21:23:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 21:24:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37fcfb41-2665-d07a-f35c-52379a78e2ff (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa840400, cur 1539577491 expire 1539577341 last 1539577264 Oct 14 21:24:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 21:26:07 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 21:35:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 14 21:35:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 21:36:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20efd741-31c1-b4ef-70fa-dca96a1dcc12 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dab42c000, cur 1539578179 expire 1539578029 last 1539577952 Oct 14 21:36:19 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 21:54:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42f40dca-1df7-0ebc-f13e-92a24c792b59 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807930a6c00, cur 1539579254 expire 1539579104 last 1539579027 Oct 14 21:54:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 21:55:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 21:55:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 21:59:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 22:03:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 22:03:34 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 22:03:34 oak-md1-s2 kernel: Lustre: 10495:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 22:07:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client df27669a-9271-042e-86bb-6e56a1dc0eb8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881af267dc00, cur 1539580075 expire 1539579925 last 1539579848 Oct 14 22:07:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 22:10:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 22:10:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 22:20:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0cade7a5-6db2-d7f9-f63d-0104bf19f0be (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041f132800, cur 1539580847 expire 1539580697 last 1539580620 Oct 14 22:20:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 22:27:19 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 22:27:19 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 22:29:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 22:33:03 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 22:33:03 oak-md1-s2 kernel: Lustre: 10635:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 22:33:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client af504a96-e73d-cf1c-4b55-2f5b5618b476 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e87512400, cur 1539581615 expire 1539581465 last 1539581388 Oct 14 22:37:39 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 22:41:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 22:41:59 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 22:42:11 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 22:46:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1f3393cf-a60a-be6a-0a85-1549738eaeba (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800561bf000, cur 1539582381 expire 1539582231 last 1539582154 Oct 14 22:46:21 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 22:55:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 22:55:02 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 22:59:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9a746a00-c60a-0aee-77e8-a398ac79cda7 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185e76d000, cur 1539583162 expire 1539583012 last 1539582935 Oct 14 22:59:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 23:03:16 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 23:03:16 oak-md1-s2 kernel: Lustre: 10544:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 23:04:15 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 23:04:15 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 14 23:08:15 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 23:08:15 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 23:10:30 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 23:10:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Oct 14 23:10:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 23:15:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7c925cf-d681-520f-3ef9-ef671f2d7078 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809c7569800, cur 1539584140 expire 1539583990 last 1539583913 Oct 14 23:15:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 23:16:14 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 23:16:14 oak-md1-s2 kernel: Lustre: 5976:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 14 23:22:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 23:22:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 14 23:26:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a489d2d-c6bd-c1db-34db-c0efaea2047f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817f072a800, cur 1539584772 expire 1539584622 last 1539584545 Oct 14 23:37:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 14 23:37:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 14 23:41:13 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 23:41:13 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 14 23:41:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d5ee89d3-7191-53bb-817d-ecf1e6fff7ad (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a66cdac00, cur 1539585681 expire 1539585531 last 1539585454 Oct 14 23:41:21 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 14 23:51:43 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 14 23:51:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6806a4de-859d-a5fe-4227-efa3acdad1dc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88059c29b800, cur 1539586304 expire 1539586154 last 1539586077 Oct 14 23:52:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 14 23:52:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 00:00:35 oak-md1-s2 kernel: Lustre: 10546:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 00:02:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9310eb3c-9252-3147-8e7a-a79e6002ca6f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afd000, cur 1539586931 expire 1539586781 last 1539586704 Oct 15 00:02:11 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 00:02:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 00:02:28 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 00:12:58 oak-md1-s2 kernel: Lustre: 4883:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 00:13:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 00:13:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 00:14:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b7ce3c6-3146-c030-ff46-392fade6533d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881734bcec00, cur 1539587657 expire 1539587507 last 1539587430 Oct 15 00:14:17 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 00:24:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 00:24:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 00:25:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a5eb339c-0587-3a8c-36a3-ca6ffacb0428 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881898b32800, cur 1539588300 expire 1539588150 last 1539588073 Oct 15 00:25:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 00:35:41 oak-md1-s2 kernel: Lustre: 10555:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 00:36:06 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 00:36:06 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 15 00:36:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 00:36:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 00:37:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f88bcb0a-198d-1d24-5a67-6986fbeda33d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880012c31400, cur 1539589032 expire 1539588882 last 1539588805 Oct 15 00:37:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 00:39:11 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 00:39:11 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 15 00:46:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 00:46:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 00:50:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f14a3b4-f775-34e7-d487-9c470cf0df0e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bd64d400, cur 1539589841 expire 1539589691 last 1539589614 Oct 15 00:50:41 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 00:53:20 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 00:53:20 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 15 00:58:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 00:58:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 01:04:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2db7600-2550-ea42-de04-d8455e66c824 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800178f9800, cur 1539590684 expire 1539590534 last 1539590457 Oct 15 01:04:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 01:10:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 01:10:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 01:20:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bae7c73-79e1-6ff5-567f-cc8d2e017ad2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe774d400, cur 1539591627 expire 1539591477 last 1539591400 Oct 15 01:20:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 01:23:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 01:23:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 01:33:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3bfe3555-c22d-7710-262d-dd4d0a4bd083 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880052c85400, cur 1539592415 expire 1539592265 last 1539592188 Oct 15 01:33:35 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 01:34:50 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 01:38:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 01:38:22 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 01:53:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 15 01:53:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 81fdf206-229b-96e9-5677-30186a4ddcb9 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac42a000, cur 1539593627 expire 1539593477 last 1539593400 Oct 15 01:53:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 01:57:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6ed4d33b-4030-aaf9-bdd0-d165c10eccf1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186c674400, cur 1539593866 expire 1539593716 last 1539593639 Oct 15 02:03:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 208dc229-120b-fb25-969a-4251ce07291e (at 10.9.113.2@o2ib4) Oct 15 02:03:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 02:03:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 26fb6d7c-cdf0-ba01-b8f1-2fd0f4e37333 (at 10.9.113.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005771bc00, cur 1539594226 expire 1539594076 last 1539593999 Oct 15 02:09:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59172b13-5eba-24d1-a747-44bac942a31f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b291d3c00, cur 1539594551 expire 1539594401 last 1539594324 Oct 15 02:15:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 02:15:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 02:23:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ce6c240-30bb-9ade-ac4f-f2197d650cbf (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af1800, cur 1539595388 expire 1539595238 last 1539595161 Oct 15 02:23:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 02:28:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80cb5673-ff2c-3087-050d-7790061c1bd0 (at 10.8.6.27@o2ib6) Oct 15 02:28:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 02:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a930ecb-6bff-f4f7-2f37-06c39c97e5a2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da1081400, cur 1539596087 expire 1539595937 last 1539595860 Oct 15 02:34:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 02:38:26 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 02:42:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 02:42:19 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 02:46:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8169e3c9-81f0-4a13-7ec9-72245af1db9b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c5f6ab800, cur 1539596766 expire 1539596616 last 1539596539 Oct 15 02:46:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 02:56:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 02:56:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 03:00:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4189d9fb-8f76-21e1-584f-afb4c4ba8d1b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88186c672800, cur 1539597601 expire 1539597451 last 1539597374 Oct 15 03:00:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 03:13:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 03:18:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe4c76ce-8555-2802-ccbe-97669d413cce (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daca14000, cur 1539598689 expire 1539598539 last 1539598462 Oct 15 03:28:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 03:28:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 03:32:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1184db8d-f592-13e5-0c57-efb3675c065d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da97ec800, cur 1539599564 expire 1539599414 last 1539599337 Oct 15 03:32:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 03:40:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 03:40:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 03:45:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c60b8bb6-4ca1-2c01-4ed2-c99b7c4272aa (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b7ae4e000, cur 1539600352 expire 1539600202 last 1539600125 Oct 15 03:45:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 03:55:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 03:55:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 03:59:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b7d026a-851b-2653-15db-5fb81b6d7ac8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c7fa23400, cur 1539601192 expire 1539601042 last 1539600965 Oct 15 03:59:52 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 04:06:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 04:06:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 04:11:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ee2d33a-d386-63b9-5091-a44ab3714ccc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a30f000, cur 1539601884 expire 1539601734 last 1539601657 Oct 15 04:11:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 04:17:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 04:17:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 04:26:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e37867d-aba4-b818-d194-5371175ad795 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88127d2e2400, cur 1539602794 expire 1539602644 last 1539602567 Oct 15 04:26:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 04:34:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 04:34:36 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 04:40:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 69f6b480-5630-692a-9808-1ac88df85f7d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ffae97c00, cur 1539603604 expire 1539603454 last 1539603377 Oct 15 04:40:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 04:48:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 04:48:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 04:52:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55a2509d-2c8d-d110-a9c1-f175dd8ffa9a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808552a1800, cur 1539604331 expire 1539604181 last 1539604104 Oct 15 04:52:11 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 04:59:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 04:59:46 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 05:03:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3d208d9a-b52f-7e14-6bf7-5ac39d993911 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db070a000, cur 1539604983 expire 1539604833 last 1539604756 Oct 15 05:03:03 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 05:10:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 05:10:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 05:13:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce50b937-2a68-cbe6-360b-50abab2a5307 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a1ee31800, cur 1539605627 expire 1539605477 last 1539605400 Oct 15 05:13:47 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 05:23:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 05:23:25 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 05:24:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4264e869-ec78-d144-291b-2d09296dec4f (at 10.8.10.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d72e37800, cur 1539606259 expire 1539606109 last 1539606032 Oct 15 05:24:19 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 05:36:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 36aa491c-4ed0-e016-5bb6-cb55574f144e (at 10.8.10.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881beea6d000, cur 1539606976 expire 1539606826 last 1539606749 Oct 15 05:36:16 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 05:37:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 05:37:36 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 15 05:46:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc5e6c12-691d-909c-acaf-99f9ba628b05 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f3ccfd000, cur 1539607577 expire 1539607427 last 1539607350 Oct 15 05:46:17 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 05:48:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 05:48:15 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 05:57:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 67991cf9-8f20-8a15-be63-9dcbb0e15077 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807fa9ab800, cur 1539608239 expire 1539608089 last 1539608012 Oct 15 05:57:19 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 05:59:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 05:59:27 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:10:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 67cda806-06d5-f035-f392-293696470291 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e75236400, cur 1539609058 expire 1539608908 last 1539608831 Oct 15 06:10:58 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:12:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 06:12:43 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:22:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cb5c857f-f805-a5e8-97a2-2ecda3145f52 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d0947c400, cur 1539609776 expire 1539609626 last 1539609549 Oct 15 06:22:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:23:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 06:23:32 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:33:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6bcc4abb-b73b-46fd-cc25-cf0e7714081a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eb981e800, cur 1539610423 expire 1539610273 last 1539610196 Oct 15 06:33:43 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:34:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 15 06:34:41 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 06:40:34 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 06:44:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b4c5b445-7672-08f0-c7c1-fd9bc304ff72 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9c800, cur 1539611055 expire 1539610905 last 1539610828 Oct 15 06:44:15 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:47:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 06:47:02 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:54:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3f236dab-600f-c798-781e-8600d226a8fa (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880860fa3800, cur 1539611686 expire 1539611536 last 1539611459 Oct 15 06:54:46 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 06:59:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 06:59:14 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 07:09:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc8d1616-05be-38b5-98d4-a95216efcede (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818973fa800, cur 1539612578 expire 1539612428 last 1539612351 Oct 15 07:09:38 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 07:12:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 07:12:10 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 07:21:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b573e76a-ee3e-277b-5eba-0151510acec6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe1343c00, cur 1539613294 expire 1539613144 last 1539613067 Oct 15 07:21:34 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 07:25:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 07:25:21 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 07:36:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49a41bf6-9bf5-2119-e2ee-1a7492fd3488 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b5f37a800, cur 1539614176 expire 1539614026 last 1539613949 Oct 15 07:36:16 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 07:37:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 07:37:02 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 07:47:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 07:47:47 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 07:48:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b634cd27-6938-e193-face-ebf7d2cbf99b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4b596800, cur 1539614897 expire 1539614747 last 1539614670 Oct 15 07:48:17 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 07:58:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 07:58:07 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 07:58:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 078215ef-bf2d-a7ba-4bea-c412aebda19f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88020cea7000, cur 1539615500 expire 1539615350 last 1539615273 Oct 15 07:58:20 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 08:04:03 oak-md1-s2 kernel: Lustre: 10661:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539615836/real 1539615836] req@ffff8813fbff6000 x1610632726956880/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539615843 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 15 08:04:03 oak-md1-s2 kernel: Lustre: 10661:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 15 08:04:10 oak-md1-s2 kernel: Lustre: 10661:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539615843/real 1539615843] req@ffff8813fbff6000 x1610632726956880/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539615850 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:04:17 oak-md1-s2 kernel: Lustre: 10661:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539615850/real 1539615850] req@ffff8813fbff6000 x1610632726956880/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539615857 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:04:24 oak-md1-s2 kernel: Lustre: 10661:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539615857/real 1539615857] req@ffff8813fbff6000 x1610632726956880/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539615864 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:10:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 08:10:52 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 08:11:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b842bbaf-a207-8eef-0d52-50aad13afedb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe7fa3c00, cur 1539616292 expire 1539616142 last 1539616065 Oct 15 08:11:32 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 08:21:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 08:21:22 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 08:21:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3087108-859c-b00a-758d-6176aa566a0b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805d4fcdc00, cur 1539616907 expire 1539616757 last 1539616680 Oct 15 08:21:47 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 08:23:34 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539617007/real 1539617007] req@ffff880139d4ad00 x1610632727353408/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539617014 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 15 08:23:42 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539617014/real 1539617014] req@ffff880139d4ad00 x1610632727353408/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539617021 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:23:49 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539617022/real 1539617022] req@ffff880139d4ad00 x1610632727353408/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539617029 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:23:56 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539617029/real 1539617029] req@ffff880139d4ad00 x1610632727353408/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539617036 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:24:10 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539617043/real 1539617043] req@ffff880139d4ad00 x1610632727353408/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539617050 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:24:10 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 15 08:24:31 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539617064/real 1539617064] req@ffff880139d4ad00 x1610632727353408/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539617071 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 08:24:31 oak-md1-s2 kernel: Lustre: 4300:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 15 08:33:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 08:33:29 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 08:33:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 23b8149f-b548-3600-1b7d-bb0bd95ceb0b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880adae74400, cur 1539617623 expire 1539617473 last 1539617396 Oct 15 08:33:43 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 08:43:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 08:43:39 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 08:44:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 08b979e6-b923-98d4-b267-1fd829622046 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881174318000, cur 1539618262 expire 1539618112 last 1539618035 Oct 15 08:44:22 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 08:55:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 08:55:07 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 15 08:55:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d5d29271-85a6-b211-66a4-a972c734878a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811eab21000, cur 1539618910 expire 1539618760 last 1539618683 Oct 15 08:55:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 09:07:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 09:07:01 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 09:07:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9963ddcd-83fa-c166-4067-9911f26e0c1c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea8ef9000, cur 1539619645 expire 1539619495 last 1539619418 Oct 15 09:07:25 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 09:22:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb3a9720-9f8f-6527-16bd-e38b087e2ada (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daab0c800, cur 1539620551 expire 1539620401 last 1539620324 Oct 15 09:22:31 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 09:22:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 09:22:59 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 09:27:29 oak-md1-s2 kernel: Lustre: DEBUG MARKER: Mon Oct 15 09:27:29 2018 Oct 15 09:30:35 oak-md1-s2 kernel: Lustre: 10489:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 09:33:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1165a73a-eb33-87fa-cab2-c7677defacd3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809a6febc00, cur 1539621207 expire 1539621057 last 1539620980 Oct 15 09:39:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 15 09:39:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 09:48:19 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 09:52:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 09:55:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5606c7d0-f9d6-9218-4bba-4dc2c551c083 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041fb38c00, cur 1539622549 expire 1539622399 last 1539622322 Oct 15 10:01:22 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 10:01:22 oak-md1-s2 kernel: Lustre: 4881:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 15 10:08:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 42aec749-f14b-cdba-dee0-093f0c0fe4e9 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ce8789400, cur 1539623329 expire 1539623179 last 1539623102 Oct 15 10:18:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 10:22:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce19a03c-5523-c69b-638b-1b5361c2f231 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880779d3e800, cur 1539624129 expire 1539623979 last 1539623902 Oct 15 10:25:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 10:29:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c4472ca0-7f4c-6057-0e95-2900b7a137b2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804bc21c800, cur 1539624570 expire 1539624420 last 1539624343 Oct 15 10:37:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 10:41:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c91963c1-3155-5349-eca4-a965555a5abf (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b66a27400, cur 1539625266 expire 1539625116 last 1539625039 Oct 15 10:45:23 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 10:49:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Oct 15 10:49:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ccbf7ed-9c42-d656-45a7-d15626dabc81 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b28fde800, cur 1539625772 expire 1539625622 last 1539625545 Oct 15 10:50:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9961d22d-5120-725d-7c36-e14e409ff7b8 (at 10.8.20.15@o2ib6) in 179 seconds. I think it's dead, and I am evicting it. exp ffff881fe7749800, cur 1539625848 expire 1539625698 last 1539625669 Oct 15 10:55:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2785325-c043-7d13-da05-16563fa6a836 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b95224000, cur 1539626107 expire 1539625957 last 1539625880 Oct 15 11:00:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Oct 15 11:00:47 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 11:01:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 21cd1505-c243-3fb0-3f43-4ae1a0484959 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882015c1b400, cur 1539626502 expire 1539626352 last 1539626275 Oct 15 11:01:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 11:07:10 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 11:07:26 oak-md1-s2 kernel: Lustre: 10559:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 11:07:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ff5e34c0-673d-2a47-78df-3fdb0b995859 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880392e3f000, cur 1539626855 expire 1539626705 last 1539626628 Oct 15 11:07:35 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 11:11:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 15 11:11:10 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 11:13:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c62910b1-d1de-6ccb-20cd-2dfe862d59d2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881906263400, cur 1539627215 expire 1539627065 last 1539626988 Oct 15 11:13:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 11:26:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 15 11:26:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 11:37:54 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 11:40:50 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 11:42:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 11:45:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8697d421-8f73-f46f-9ae6-e9ee7da00f49 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88057bcda000, cur 1539629137 expire 1539628987 last 1539628910 Oct 15 11:50:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 13df4cc0-c924-99d9-37ea-a64910c42703 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880067359400, cur 1539629422 expire 1539629272 last 1539629195 Oct 15 11:50:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 11:51:10 oak-md1-s2 kernel: Lustre: 10642:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 11:56:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Oct 15 11:56:02 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 11:56:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 009e9d68-a236-490e-c912-7b2408eefc40 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880343111400, cur 1539629787 expire 1539629637 last 1539629560 Oct 15 12:02:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de11adef-e445-1f56-2558-284677b3d9e2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881551b39000, cur 1539630143 expire 1539629993 last 1539629916 Oct 15 12:02:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 12:05:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client acf20e58-0348-2182-7b5c-bfd5f6510fd6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184d7afc00, cur 1539630321 expire 1539630171 last 1539630094 Oct 15 12:06:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 805736a5-9324-0e02-53a1-0de6644af99c (at 10.8.18.34@o2ib6) in 185 seconds. I think it's dead, and I am evicting it. exp ffff8800217c3400, cur 1539630397 expire 1539630247 last 1539630212 Oct 15 12:06:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Oct 15 12:06:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 12:07:06 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 12:07:06 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 15 12:11:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6729bc1e-90ff-9918-f6b3-fe07e4779911 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe7fa5800, cur 1539630714 expire 1539630564 last 1539630487 Oct 15 12:11:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 12:16:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ab9735f4-203c-d387-3603-a6aa508665ce (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103aa1a000, cur 1539631008 expire 1539630858 last 1539630781 Oct 15 12:16:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 12:18:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Oct 15 12:18:54 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 12:24:08 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 12:24:08 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 15 12:25:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28302a4d-26a0-37a0-4c09-3ccea89b5a88 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa225800, cur 1539631523 expire 1539631373 last 1539631296 Oct 15 12:25:23 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 12:29:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 30680104-7926-0ac0-e3ba-288b7516aa44 (at 10.8.18.34@o2ib6) Oct 15 12:29:05 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 12:38:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7a7a9085-f00e-5352-1c8e-44601c4eab9c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003a19f800, cur 1539632297 expire 1539632147 last 1539632070 Oct 15 12:38:17 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 12:41:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 12:41:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 12:50:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20551f6f-be57-70da-5815-802b1e931b50 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88020cea4c00, cur 1539633039 expire 1539632889 last 1539632812 Oct 15 12:50:39 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 12:54:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 12:54:48 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 12:57:07 oak-md1-s2 kernel: Lustre: 10482:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 13:03:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b114632b-2d1e-2635-006c-26d070bf7c5e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff4388400, cur 1539633790 expire 1539633640 last 1539633563 Oct 15 13:03:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 13:06:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 13:06:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 13:15:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e04a1e7-a9de-5145-d3cf-4a895ea50033 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee1a07800, cur 1539634555 expire 1539634405 last 1539634328 Oct 15 13:15:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 13:16:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 13:16:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 13:18:52 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 13:29:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 13:29:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 13:30:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 13944de6-f2a6-f54f-5d49-5096757aba0c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f2afc00, cur 1539635409 expire 1539635259 last 1539635182 Oct 15 13:30:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 13:51:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c79acf1-9e36-2f77-eaa3-cd9380650982 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185c29b400, cur 1539636692 expire 1539636542 last 1539636465 Oct 15 13:51:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 13:52:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5d7ad9df-6849-08bb-cc93-09c9a4d0029d (at 10.8.18.34@o2ib6) in 198 seconds. I think it's dead, and I am evicting it. exp ffff880013a64800, cur 1539636768 expire 1539636618 last 1539636570 Oct 15 13:53:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 13:53:11 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 13:54:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 13:54:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 13:57:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 13:57:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5c5c51ce-8800-ee45-6e69-a8f7e83a3416 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b1ec0cc00, cur 1539637052 expire 1539636902 last 1539636825 Oct 15 13:57:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 14:04:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 14:04:18 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 14:04:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ca6b4fa-e237-e951-b405-9234a5cd86f1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880020d03400, cur 1539637494 expire 1539637344 last 1539637267 Oct 15 14:04:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 14:18:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 14:18:07 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 14:18:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6afec38-7eb4-79c8-57dc-9de943ab743a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809a6fecc00, cur 1539638329 expire 1539638179 last 1539638102 Oct 15 14:18:49 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 14:29:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 14:29:58 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 14:30:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a5ad71b-302e-7f95-56ae-554dfe0a9e77 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814f0360c00, cur 1539639022 expire 1539638872 last 1539638795 Oct 15 14:30:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 14:41:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a8ce4562-572a-7e13-e916-0d2914ca86b1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea8c46c00, cur 1539639707 expire 1539639557 last 1539639480 Oct 15 14:41:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 14:41:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 14:41:54 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 14:53:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 14:53:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 14:55:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79966a65-2203-3722-24d0-a7dac2e46335 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851230400, cur 1539640548 expire 1539640398 last 1539640321 Oct 15 14:55:48 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 15:04:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 15:04:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 15:07:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 618cda2d-9268-84e6-6d6c-5e5e121893f8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b99267400, cur 1539641243 expire 1539641093 last 1539641016 Oct 15 15:07:23 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 15:15:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 15:15:33 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 15:20:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 039b7d7c-c4ae-af16-db06-d84ad3e48b02 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca1000, cur 1539642036 expire 1539641886 last 1539641809 Oct 15 15:20:36 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 15:26:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 15:26:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 15:31:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 153c5a5f-11f7-c68d-a39b-f0be6db02774 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c1f342400, cur 1539642671 expire 1539642521 last 1539642444 Oct 15 15:31:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 15:41:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5274e48c-47e4-c22a-3c7d-f3d23f4ff3ce (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88189aa8f000, cur 1539643276 expire 1539643126 last 1539643049 Oct 15 15:41:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 15:44:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 15:44:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 15:54:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74f627b6-121d-90b5-0a51-d7cbfdc37b08 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d252ec400, cur 1539644064 expire 1539643914 last 1539643837 Oct 15 15:54:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 15:56:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 15:56:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 16:08:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a05b4aa4-1d0b-776a-d869-890373a81d55 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc230dc00, cur 1539644889 expire 1539644739 last 1539644662 Oct 15 16:08:09 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 16:08:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 16:08:11 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 16:18:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 16:18:53 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 16:19:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f825989e-e73a-505a-364f-63b9e1a5d54a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800585fe400, cur 1539645541 expire 1539645391 last 1539645314 Oct 15 16:19:01 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 16:31:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 16:31:21 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 16:32:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1c92a921-883f-7266-c041-f8762b446e46 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88145768d000, cur 1539646358 expire 1539646208 last 1539646131 Oct 15 16:32:38 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 16:42:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 16:42:11 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 16:42:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5eb0438d-4ed8-1403-69dc-b25219004da2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88152c641c00, cur 1539646974 expire 1539646824 last 1539646747 Oct 15 16:42:54 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 16:52:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 16:52:51 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 16:57:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 674caee0-4757-e6b0-c400-0883ba691e2d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882016779800, cur 1539647832 expire 1539647682 last 1539647605 Oct 15 16:57:12 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 17:04:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 17:04:20 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 17:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2360cdbe-07fc-3037-ee97-1071fcfae712 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817e0f11800, cur 1539648542 expire 1539648392 last 1539648315 Oct 15 17:09:02 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 17:14:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 17:14:33 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 17:19:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 80cb6324-8f80-50a1-0192-639768f9df46 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058c465c00, cur 1539649176 expire 1539649026 last 1539648949 Oct 15 17:19:36 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 17:24:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 17:24:43 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 17:29:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f062023c-d830-399d-2b3e-df5d57a12324 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfefc3000, cur 1539649785 expire 1539649635 last 1539649558 Oct 15 17:29:45 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 17:39:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 17:39:29 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 17:42:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7e06742-1acf-64f4-8554-22cd73764028 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e210f400, cur 1539650525 expire 1539650375 last 1539650298 Oct 15 17:42:05 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 17:49:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 17:49:38 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 17:53:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cf06f769-606c-63d4-52bb-ba448a6ab088 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb98000, cur 1539651239 expire 1539651089 last 1539651012 Oct 15 17:53:59 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 18:02:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 18:02:21 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 18:08:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 12e6a282-b191-e38d-b2f9-c792f6600a06 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881540790800, cur 1539652088 expire 1539651938 last 1539651861 Oct 15 18:08:08 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 15 18:09:09 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539652142/real 1539652142] req@ffff8811c2b4f200 x1610632763438992/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539652149 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 15 18:09:16 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539652149/real 1539652149] req@ffff8811c2b4f200 x1610632763438992/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539652156 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 18:09:31 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539652163/real 1539652163] req@ffff8811c2b4f200 x1610632763438992/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539652170 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 18:09:31 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 15 18:09:52 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539652185/real 1539652185] req@ffff8811c2b4f200 x1610632763438992/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539652192 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 18:09:52 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 15 18:10:34 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539652227/real 1539652227] req@ffff8811c2b4f200 x1610632763438992/t0(0) o104->oak-MDT0000@10.8.17.26@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539652234 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 15 18:10:34 oak-md1-s2 kernel: Lustre: 10597:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 15 18:13:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 18:13:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 18:20:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4a1dcb26-4185-b9e5-3cf0-41bfec20efbe (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881495248400, cur 1539652816 expire 1539652666 last 1539652589 Oct 15 18:20:16 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 18:23:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 18:23:59 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 18:30:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e2bc425-beaa-29b6-6bf5-c1950f8260c5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf1ea6400, cur 1539653442 expire 1539653292 last 1539653215 Oct 15 18:30:42 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 18:35:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 18:35:21 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 18:44:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec35b3b0-2235-94f4-1fb0-b8d467e58092 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2112400, cur 1539654253 expire 1539654103 last 1539654026 Oct 15 18:44:13 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 18:48:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 18:48:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 18:56:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client faa58c43-ed65-c3cf-b920-abfb8ff76fed (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882016779000, cur 1539655004 expire 1539654854 last 1539654777 Oct 15 18:56:44 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 19:00:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 19:00:36 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 19:09:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3fecd2e1-a89b-9796-ab2a-829badbd815a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880727403c00, cur 1539655749 expire 1539655599 last 1539655522 Oct 15 19:09:09 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 15 19:13:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 15 19:13:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 15 19:21:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9309454d-4370-e782-1ad2-2559acaf9be8 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a71ac00, cur 1539656474 expire 1539656324 last 1539656247 Oct 15 19:21:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 19:27:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 15 19:27:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 19:42:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 27db37cf-f664-fb46-4843-c0e59b345349 (at 10.8.14.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185baec800, cur 1539657753 expire 1539657603 last 1539657526 Oct 15 19:42:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 20:02:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 20:06:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2082cc0c-671b-7ec1-164a-449e606a14b0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203a2f7400, cur 1539659168 expire 1539659018 last 1539658941 Oct 15 20:06:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 20:11:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb955935-876a-105f-7062-120f7fd3f8ac (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880810d09000, cur 1539659479 expire 1539659329 last 1539659252 Oct 15 20:30:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 15 20:31:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2bec3668-720e-29ca-0a79-72e36893a110 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bf657b000, cur 1539660689 expire 1539660539 last 1539660462 Oct 15 20:37:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 15 20:41:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f6a8d28e-6d7b-ee43-3944-fd2fff24fdbc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff0b37000, cur 1539661265 expire 1539661115 last 1539661038 Oct 15 21:13:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3021e943-12e8-9f36-99ac-f1203c737797 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818692dd800, cur 1539663187 expire 1539663037 last 1539662960 Oct 15 21:16:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 15 21:24:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 15 21:24:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef3a2e3f-c2dc-d5ec-674b-74e359e54f43 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818feaff800, cur 1539663887 expire 1539663737 last 1539663660 Oct 15 21:36:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 14c15579-c854-deec-5bb9-39998fd0aeb0 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacc5e000, cur 1539664612 expire 1539664462 last 1539664385 Oct 15 21:41:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 15 21:42:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ade835f9-754e-f9cb-a7bf-2720e59e168e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88059c299c00, cur 1539664955 expire 1539664805 last 1539664728 Oct 15 21:42:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 21:45:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 15 21:46:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 76af6ff8-60be-9604-aeb0-52a94ed5d152 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef094000, cur 1539665183 expire 1539665033 last 1539664956 Oct 15 21:50:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7aa48738-dd1a-c0a2-b5ac-fdfd7606df42 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe7fa0400, cur 1539665420 expire 1539665270 last 1539665193 Oct 15 21:54:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 21:58:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 21:58:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 222485d5-a9a5-b3f6-467d-8315267ba4b1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ffae96800, cur 1539665934 expire 1539665784 last 1539665707 Oct 15 22:02:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 22:04:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43be68e4-05e2-4302-7c66-5a2fca9b947e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e7a3400, cur 1539666243 expire 1539666093 last 1539666016 Oct 15 22:05:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2216c46f-37cc-15c9-bdf4-e23b162fe8d1 (at 10.8.20.15@o2ib6) in 151 seconds. I think it's dead, and I am evicting it. exp ffff8818692dd400, cur 1539666319 expire 1539666169 last 1539666168 Oct 15 22:05:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 22:09:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 22:09:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 57807381-f2ca-fef1-eb79-10673d3d465a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c5bb8d400, cur 1539666585 expire 1539666435 last 1539666358 Oct 15 22:16:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 22:17:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b8daa25-531d-0b78-01fc-9dd15c6bfabf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf27e800, cur 1539667071 expire 1539666921 last 1539666844 Oct 15 22:24:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 22:25:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 91de68d6-e9a3-aab9-1d46-2beed7d1f437 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815e56f9400, cur 1539667503 expire 1539667353 last 1539667276 Oct 15 22:31:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7370e59c-284a-b915-0b61-85f8d6fcb80d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881540795000, cur 1539667912 expire 1539667762 last 1539667685 Oct 15 22:31:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 15 23:06:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a51aad02-ee21-eef4-87e6-bca324375956 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b42aa0c00, cur 1539669968 expire 1539669818 last 1539669741 Oct 15 23:07:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 23:07:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 15 23:13:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 23:13:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 18907570-bd76-4c34-9505-04606c21dfc3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee7554000, cur 1539670419 expire 1539670269 last 1539670192 Oct 15 23:16:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 23:17:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c281a6e8-f2e8-8a2a-270c-282fee562d07 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102a2fa400, cur 1539670656 expire 1539670506 last 1539670429 Oct 15 23:49:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 23:50:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 526cc67d-e79c-ec4d-5f98-6d38399dc37f (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc317cc00, cur 1539672604 expire 1539672454 last 1539672377 Oct 15 23:51:53 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 15 23:57:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 15 23:58:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db1b3be2-485e-73ed-2c3d-b6bc0090ebf8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d094b8800, cur 1539673085 expire 1539672935 last 1539672858 Oct 15 23:58:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 00:00:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 00:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8360ecef-17ec-81d9-4a35-864a8649a2eb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8aff000, cur 1539673320 expire 1539673170 last 1539673093 Oct 16 00:06:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 00:06:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 00:10:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8982a894-6a13-828a-96be-c1cb1bc8a61f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881890e4f800, cur 1539673821 expire 1539673671 last 1539673594 Oct 16 00:18:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 00:18:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b798d295-8928-4ab7-020e-d436dd418153 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88058c467800, cur 1539674330 expire 1539674180 last 1539674103 Oct 16 00:19:43 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 00:36:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 00:37:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f2d0e764-d24a-c626-a69f-506a93cf7f7a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800178fa000, cur 1539675430 expire 1539675280 last 1539675203 Oct 16 00:48:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 00:52:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 232c3249-3567-312d-947a-9b510d395422 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2114c00, cur 1539676341 expire 1539676191 last 1539676114 Oct 16 00:52:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 01:03:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 01:03:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 01:06:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf0047ce-a06c-c8d7-8c2d-45ebefc84029 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ac06f8800, cur 1539677207 expire 1539677057 last 1539676980 Oct 16 01:27:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 01:27:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 01:31:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 90de2168-eead-56a4-76c8-0f2b0e1dc2f1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e61521c00, cur 1539678682 expire 1539678532 last 1539678455 Oct 16 01:31:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 01:33:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 01:37:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 409033da-4043-5a3d-1819-44099a849701 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881800344400, cur 1539679038 expire 1539678888 last 1539678811 Oct 16 01:37:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 01:41:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d1c4e314-024c-13bd-24e3-cd221ec13666 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807930a1000, cur 1539679278 expire 1539679128 last 1539679051 Oct 16 01:43:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 01:43:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 01:48:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c6d70782-9052-d1cc-0bf7-dfdfa9811bee (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ed255c00, cur 1539679722 expire 1539679572 last 1539679495 Oct 16 01:48:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 01:54:34 oak-md1-s2 kernel: Lustre: 10508:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 01:54:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 01:54:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 02:04:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb3b4c9c-cd58-04f4-0f8f-ff11625ca42d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c93ff4800, cur 1539680691 expire 1539680541 last 1539680464 Oct 16 02:04:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 02:09:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 02:09:38 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 02:17:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4fbc33dc-1982-ccef-914b-7fd1a69b04a7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa85ec00, cur 1539681451 expire 1539681301 last 1539681224 Oct 16 02:17:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 02:24:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 02:24:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 02:30:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53a75760-5064-df9d-3b62-103170335ff3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096a0f2000, cur 1539682218 expire 1539682068 last 1539681991 Oct 16 02:30:18 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 02:36:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 02:36:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 02:40:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a4e6b29-5729-d792-72ab-d8bef0f300f5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d0e236c00, cur 1539682838 expire 1539682688 last 1539682611 Oct 16 02:40:38 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 02:47:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 02:47:05 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 02:50:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e37f7516-301d-7a25-534a-f189d8b581de (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881959690000, cur 1539683452 expire 1539683302 last 1539683225 Oct 16 02:50:52 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 02:58:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 02:58:28 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 16 03:02:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f354831a-b85f-5440-c142-e08eb4307fd8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185c29a000, cur 1539684135 expire 1539683985 last 1539683908 Oct 16 03:02:15 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 16 03:08:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 03:08:45 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 16 03:13:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 527cdb39-6453-8a33-734d-bf4abed231bb (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88181977ec00, cur 1539684802 expire 1539684652 last 1539684575 Oct 16 03:13:22 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 03:20:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 03:20:26 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 03:28:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6e6fe7b8-b52b-d566-ee92-beeee2efcfb6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea84dac00, cur 1539685703 expire 1539685553 last 1539685476 Oct 16 03:28:23 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 16 03:30:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 03:30:49 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 03:39:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 10a2dc34-133b-c30b-e3eb-c0c5dc9d16e2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11867000, cur 1539686378 expire 1539686228 last 1539686151 Oct 16 03:39:38 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 03:42:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 03:42:55 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 03:50:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 10200849-0543-37b4-8b6b-3e55404d867d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee7552c00, cur 1539687052 expire 1539686902 last 1539686825 Oct 16 03:50:52 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 03:56:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 03:56:11 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 04:04:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7e719a53-8020-b502-967e-d609c3aae41e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185baef000, cur 1539687849 expire 1539687699 last 1539687622 Oct 16 04:04:09 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 04:08:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 04:08:47 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 04:16:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2700b279-e643-98c7-6fc8-ea492dc60ec4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880562b4e000, cur 1539688619 expire 1539688469 last 1539688392 Oct 16 04:16:59 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 04:19:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 04:19:04 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 04:28:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 80f56f71-4a5c-9736-5feb-2ef7a6d3f9c0 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800178fbc00, cur 1539689315 expire 1539689165 last 1539689088 Oct 16 04:28:35 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 04:30:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 04:30:46 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 16 04:39:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4cd93a8-138b-9974-4698-cb7418fde26a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881037d46c00, cur 1539689973 expire 1539689823 last 1539689746 Oct 16 04:39:33 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 04:42:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 04:42:41 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 04:45:00 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 04:50:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea5be282-b7c8-99f6-732f-494ff82cbd92 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f4800, cur 1539690644 expire 1539690494 last 1539690417 Oct 16 04:50:44 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 04:52:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 04:52:59 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 05:03:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 05:03:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 05:03:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d84c1ce-1f46-0243-58aa-bf3ed5d9132a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805adfabc00, cur 1539691432 expire 1539691282 last 1539691205 Oct 16 05:03:52 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 05:15:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 05:15:57 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 16 05:17:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2cc67b88-8847-a28e-ef4d-0425ce0d5517 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c1f344400, cur 1539692228 expire 1539692078 last 1539692001 Oct 16 05:17:08 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 16 05:29:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e4d95819-8155-8535-d643-4dc972d9528e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb87000, cur 1539692955 expire 1539692805 last 1539692728 Oct 16 05:29:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 05:30:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 05:30:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 05:41:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 05:41:39 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 05:41:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 558a0f70-a953-e2b6-8de7-9491227f8b78 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f593dc000, cur 1539693715 expire 1539693565 last 1539693488 Oct 16 05:41:55 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 05:45:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 53e5733b-f19a-61af-3462-34473914e290 (at 10.8.10.32@o2ib6) reconnecting Oct 16 05:56:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e73cb35-c8c2-2cdf-b2ac-636ab92eea90 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fea644000, cur 1539694572 expire 1539694422 last 1539694345 Oct 16 05:56:12 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 05:56:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 05:56:24 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 06:07:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7bc98a88-2d13-a286-0cb7-0840960a9f07 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272375400, cur 1539695236 expire 1539695086 last 1539695009 Oct 16 06:07:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 06:09:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 06:09:47 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 06:19:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 67bd7ada-a279-4a65-0336-30dec666b9ce (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88037d284800, cur 1539695989 expire 1539695839 last 1539695762 Oct 16 06:19:49 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 06:22:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 06:22:04 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 06:31:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a50db9fc-073b-bba7-e6a4-eda3d4c07e95 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031dc75c00, cur 1539696686 expire 1539696536 last 1539696459 Oct 16 06:31:26 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 06:32:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 06:32:22 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 06:41:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53d2421d-9543-8869-bf50-b5738c5400b8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa85ec00, cur 1539697308 expire 1539697158 last 1539697081 Oct 16 06:41:48 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 06:44:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 06:44:47 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 06:53:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8afb4aca-8544-0947-dcaa-737b715a9037 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880730189c00, cur 1539698028 expire 1539697878 last 1539697801 Oct 16 06:53:48 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 06:57:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 06:57:10 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 07:04:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7871a9ac-c1de-5f51-fdca-fa5311b5533e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817f072f000, cur 1539698657 expire 1539698507 last 1539698430 Oct 16 07:04:17 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 07:10:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 07:10:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 07:20:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 416b6f15-554e-c88c-06b2-a15df3a55e7d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d094bec00, cur 1539699600 expire 1539699450 last 1539699373 Oct 16 07:20:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 07:29:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 07:29:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 07:30:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eed7f939-c196-c74c-7a14-02c188ee6dde (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102dfeb800, cur 1539700222 expire 1539700072 last 1539699995 Oct 16 07:43:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 07:43:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ea30e10-8755-aa22-8068-66a0d36dc923 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b268ec00, cur 1539701027 expire 1539700877 last 1539700800 Oct 16 07:57:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 07:57:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 07:57:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b863e401-adab-db8b-b70d-348ddb42988f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d252ec400, cur 1539701866 expire 1539701716 last 1539701639 Oct 16 07:57:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 08:14:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 08:14:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 714bfc08-59c2-d5ce-e882-c3b7d45b2ca5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881befb82800, cur 1539702899 expire 1539702749 last 1539702672 Oct 16 08:26:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 08:26:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 08:30:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e98495c-8dcf-69ec-43e2-75552443ee46 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818feafac00, cur 1539703842 expire 1539703692 last 1539703615 Oct 16 08:30:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 08:36:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 08:36:32 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 16 08:44:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6f37c89-a6ea-e6f0-7755-cb097a47f07d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f196e2400, cur 1539704644 expire 1539704494 last 1539704417 Oct 16 08:44:04 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 16 08:51:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 08:51:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 08:56:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 559b21cd-4479-b2c6-2d0b-687f0646ceb1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ac06fa800, cur 1539705397 expire 1539705247 last 1539705170 Oct 16 08:56:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 09:07:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 09:07:58 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 09:08:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28ae8cb5-9c5d-ce50-adc9-2b5c47561e32 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ed353b000, cur 1539706100 expire 1539705950 last 1539705873 Oct 16 09:08:20 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 09:20:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4353e9b2-856a-2fee-34de-c32ac36bd5fe (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880860fa6800, cur 1539706851 expire 1539706701 last 1539706624 Oct 16 09:20:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 09:21:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 09:21:38 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 09:33:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 09:33:35 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 09:37:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7c4f7ece-382f-b6df-d485-30c730fa9591 (at 10.12.4.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88057bcdd000, cur 1539707856 expire 1539707706 last 1539707629 Oct 16 09:37:36 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 09:44:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 09:44:48 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 16 09:49:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fc24b2e2-b7c4-b47e-aeb1-3fe115fa7c18 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881816ee4800, cur 1539708548 expire 1539708398 last 1539708321 Oct 16 09:49:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 10:00:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 10:00:45 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 10:01:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f86e05b-b605-9e12-84c6-d338bc8c3a41 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041fb3f400, cur 1539709261 expire 1539709111 last 1539709034 Oct 16 10:01:01 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 10:11:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 865a8977-39fe-295c-d2c4-3d9dbb697416 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881698e95c00, cur 1539709868 expire 1539709718 last 1539709641 Oct 16 10:11:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 10:11:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 16 10:11:11 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 10:21:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98f764da-0da7-a7e0-9805-4f0fe3784158 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803e8240c00, cur 1539710478 expire 1539710328 last 1539710251 Oct 16 10:21:18 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 10:22:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 10:22:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 10:34:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b955837e-ada6-f3fc-6ed8-b9e22ff14809 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d0947ec00, cur 1539711296 expire 1539711146 last 1539711069 Oct 16 10:34:56 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 10:40:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 10:40:38 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 10:50:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 26c64de9-94d2-256b-b53b-27bdc02cab5d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272373800, cur 1539712220 expire 1539712070 last 1539711993 Oct 16 10:50:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 10:55:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 10:55:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 11:03:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9b9e8a95-f09b-12e5-6b04-630952772067 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e7a5800, cur 1539712981 expire 1539712831 last 1539712754 Oct 16 11:03:01 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 11:08:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 11:08:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 11:09:02 oak-md1-s2 kernel: Lustre: 10484:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 11:09:02 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 11:11:02 oak-md1-s2 kernel: Lustre: 10672:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 11:14:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b974d26-d035-f075-b843-9a0aa7477d0f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881855b73400, cur 1539713647 expire 1539713497 last 1539713420 Oct 16 11:14:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 11:23:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 11:23:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 11:26:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5833729a-6286-ced4-655f-3c2378105ab6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881283778800, cur 1539714413 expire 1539714263 last 1539714186 Oct 16 11:26:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 11:34:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 11:34:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 11:40:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dd2de48b-4104-0505-ae1e-13798f5e505c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d0c348c00, cur 1539715213 expire 1539715063 last 1539714986 Oct 16 11:40:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 11:45:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 11:45:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 11:51:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ac22f4d-72c4-63b8-4c5c-17103f62f689 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880819d34800, cur 1539715912 expire 1539715762 last 1539715685 Oct 16 11:51:52 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 11:58:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 11:58:14 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 12:02:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30d3e02c-5f9e-01f4-9914-2289713342b0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881beea69800, cur 1539716555 expire 1539716405 last 1539716328 Oct 16 12:02:35 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 12:09:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 12:09:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 12:13:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a3e41d8-cbff-0e70-89c5-78b9d559bbbd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daaa92c00, cur 1539717196 expire 1539717046 last 1539716969 Oct 16 12:13:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 12:20:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 12:20:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 12:23:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eab23cad-f825-d687-de3f-7eb55fb08e81 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cb2a7d000, cur 1539717839 expire 1539717689 last 1539717612 Oct 16 12:23:59 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 12:32:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 16 12:32:24 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 12:36:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8f642dc0-b6a3-758c-5f61-3e3b32ceac0b (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc7fa6000, cur 1539718617 expire 1539718467 last 1539718390 Oct 16 12:36:57 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 12:58:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 12:58:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 12:59:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7257f348-1a79-e22d-0052-f0a04fefd0fa (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4b596000, cur 1539719946 expire 1539719796 last 1539719719 Oct 16 13:03:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 371db1a8-cdf4-1a59-79a7-cd8a9de36136 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e6ba0ec00, cur 1539720197 expire 1539720047 last 1539719970 Oct 16 13:06:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0753b8c0-fb60-c33a-52ba-09c7333ad286 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880011e20400, cur 1539720367 expire 1539720217 last 1539720140 Oct 16 13:06:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 13:10:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 13:17:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 591309ae-8cb7-3d97-55c8-8b4bb9833756 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1cc00, cur 1539721049 expire 1539720899 last 1539720822 Oct 16 13:17:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 13:17:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 13:18:16 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 13:18:16 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 13:25:32 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 13:25:32 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 13:25:37 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 13:25:37 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 16 13:25:41 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 13:25:41 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 16 13:29:51 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 13:29:51 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 13:33:14 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 13:33:14 oak-md1-s2 kernel: Lustre: 10660:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 16 13:35:38 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 13:35:38 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 16 13:36:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e5b334f-139f-b0e3-7f34-78a160ec8f03 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88179c236c00, cur 1539722213 expire 1539722063 last 1539721986 Oct 16 13:36:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 13:37:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 13:37:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 13:48:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 129efff3-2fb6-1bb9-5822-6b1ef7bef3c8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881596267400, cur 1539722884 expire 1539722734 last 1539722657 Oct 16 13:48:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 13:48:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 13:48:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 14:01:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 14:01:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 14:01:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 61609e43-1988-570c-0615-3daeae51a39e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880042964400, cur 1539723714 expire 1539723564 last 1539723487 Oct 16 14:01:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 14:12:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 14:12:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 14:15:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3442a186-7a6c-1e3c-ce3e-9791c34a0075 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810166a4400, cur 1539724505 expire 1539724355 last 1539724278 Oct 16 14:15:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 14:27:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7d4433d8-921c-0a7c-3ccb-26ba1ef18d63 (at 10.8.9.4@o2ib6) Oct 16 14:27:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b8749c58-04c2-9144-5f45-bf12d00e7dfe (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1e400, cur 1539725277 expire 1539725127 last 1539725050 Oct 16 14:38:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cf8040da-16c9-425b-38fd-0a9775d25566 (at 10.9.107.10@o2ib4) Oct 16 14:38:25 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 14:45:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 305ca96f-3988-f9a7-8d7f-0db7440fc518 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880839206800, cur 1539726353 expire 1539726203 last 1539726126 Oct 16 14:45:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 14:53:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 14:53:50 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 15:01:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 655c1f68-10dc-db70-fd72-0fab0c5cdd2b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018dd74000, cur 1539727292 expire 1539727142 last 1539727065 Oct 16 15:01:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 15:13:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 15:13:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 15:13:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2f77aa1b-f93f-1107-8c58-631ce7473724 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e59b12c00, cur 1539728008 expire 1539727858 last 1539727781 Oct 16 15:24:14 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 15:24:14 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 16 15:27:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 55573e92-b0de-7f04-62fc-6867bf399d54 (at 10.9.107.11@o2ib4) Oct 16 15:27:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 15:29:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d3bd864-f532-343f-3b97-df9018789f23 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8813f383fc00, cur 1539728940 expire 1539728790 last 1539728713 Oct 16 15:29:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 15:38:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 15:38:26 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 15:47:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7a4832f-5905-8bfa-b401-7e45b7d8553a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b67287c00, cur 1539730058 expire 1539729908 last 1539729831 Oct 16 15:47:38 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 15:50:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 167059a8-887d-f282-1560-0fbcdd188dab (at 10.8.13.23@o2ib6) Oct 16 15:50:42 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 16 15:57:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33e0bc55-505d-b540-e4d2-0ac954274e60 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d252e9800, cur 1539730671 expire 1539730521 last 1539730444 Oct 16 16:15:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 16:15:36 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 16 16:16:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0287969c-436d-a1a1-7378-de41f2ac1afb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa859c00, cur 1539731765 expire 1539731615 last 1539731538 Oct 16 16:16:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 16:17:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Oct 16 16:27:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Oct 16 16:30:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 331c578d-9252-0caa-cf55-5ba0f809869d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000deb2c00, cur 1539732613 expire 1539732463 last 1539732386 Oct 16 16:30:13 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 16:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 16:34:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 16:45:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 16 16:45:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 16:46:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 991e0cf7-4781-9f71-3256-7c8c78e21d09 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881f2b7dec00, cur 1539733565 expire 1539733415 last 1539733338 Oct 16 16:46:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 16:57:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 47e4fec9-3f70-966d-e91f-dba9c2d7ca48 (at 10.8.2.13@o2ib6) reconnecting Oct 16 16:57:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd37ab47-2674-29f4-6a97-a17dda5b9850 (at 10.8.2.13@o2ib6) Oct 16 16:57:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 16:58:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 47e4fec9-3f70-966d-e91f-dba9c2d7ca48 (at 10.8.2.13@o2ib6) reconnecting Oct 16 16:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b2ffc872-1790-1175-6a9b-778c4c6d2881 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bfa329c00, cur 1539734312 expire 1539734162 last 1539734085 Oct 16 17:09:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 30b2c043-d0ed-b558-da31-f665d108f012 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881734bcf000, cur 1539734956 expire 1539734806 last 1539734729 Oct 16 17:10:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 16 17:10:25 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 17:19:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7aaf3eee-f739-e6f5-89cf-0379b31be0e3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba7800, cur 1539735594 expire 1539735444 last 1539735367 Oct 16 17:19:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 17:23:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 17:23:50 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 17:34:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 16 17:34:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 17:38:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5d5887c9-24c8-c258-1c74-03c5978917f4 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfefc1800, cur 1539736738 expire 1539736588 last 1539736511 Oct 16 17:38:58 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 17:48:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 17:48:16 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 17:49:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc74ba2f-9e44-fd42-13c4-62b06abe2458 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f56b13400, cur 1539737351 expire 1539737201 last 1539737124 Oct 16 17:49:11 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 18:04:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9525fcc-f24c-f41d-57a1-9a8cd234e985 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb2138000, cur 1539738283 expire 1539738133 last 1539738056 Oct 16 18:04:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 18:05:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 18:05:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 18:15:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 18:15:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 18:15:49 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 18:15:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 14b1cc76-2ec6-6065-8cc9-a4ae62b4eb7e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815bc234000, cur 1539738954 expire 1539738804 last 1539738727 Oct 16 18:15:54 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 18:26:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 16 18:26:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 18:26:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6abfde73-01ef-ec9a-922b-1c0353521e4b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080dbc6000, cur 1539739613 expire 1539739463 last 1539739386 Oct 16 18:26:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 18:37:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd1304f2-c3c8-ce34-3610-2c3070d436f1 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809762ef800, cur 1539740228 expire 1539740078 last 1539740001 Oct 16 18:37:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 18:42:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 18:42:46 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 16 18:52:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 91913737-9224-4788-5509-74de24c14c76 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881852af0c00, cur 1539741144 expire 1539740994 last 1539740917 Oct 16 18:52:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 18:57:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 19:05:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 434ae0fc-59b7-13f9-972d-4477b9aecd87 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005c89a000, cur 1539741905 expire 1539741755 last 1539741678 Oct 16 19:10:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 19:10:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 19:17:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6207c2a7-88af-3c65-ffd0-54854abd3c8b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa905000, cur 1539742630 expire 1539742480 last 1539742403 Oct 16 19:17:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 19:21:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 19:21:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 19:27:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 495ae47c-b12a-1e3a-ef23-822255e2d7cf (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e71c8800, cur 1539743237 expire 1539743087 last 1539743010 Oct 16 19:27:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 19:38:40 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 19:38:40 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 19:41:49 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 19:42:52 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 19:42:52 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 19:45:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 19:51:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 19:51:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 45a2933e-4edb-faf2-0ac2-6d3bad60fc5b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880963361800, cur 1539744705 expire 1539744555 last 1539744478 Oct 16 19:58:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75a5dac1-1313-c2a8-bdd6-593720e4bd5f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f2afc00, cur 1539745097 expire 1539744947 last 1539744870 Oct 16 19:59:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 20:03:48 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 20:03:48 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 20:07:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3304c4b9-4457-f68b-2182-9c6acf64706a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da7a70800, cur 1539745642 expire 1539745492 last 1539745415 Oct 16 20:11:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 20:11:47 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 20:11:47 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 16 20:18:32 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 20:19:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c057ad4-0893-778c-b0e4-e66e95bd2e0a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805a616f000, cur 1539746373 expire 1539746223 last 1539746146 Oct 16 20:19:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 20:23:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 20:23:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 20:30:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 23cc7128-9e3a-0163-97dc-99aaaba90593 (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815bc236400, cur 1539747054 expire 1539746904 last 1539746827 Oct 16 20:30:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 20:36:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 20:36:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 20:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4c0353b-1c6d-b3a6-b6e4-447d32653919 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88068aa2c400, cur 1539747789 expire 1539747639 last 1539747562 Oct 16 20:43:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 20:47:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 20:47:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 20:54:34 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 20:54:34 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 20:55:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de0562e0-b9b8-6d46-2cfa-d118289f9f66 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809d9ca4400, cur 1539748547 expire 1539748397 last 1539748320 Oct 16 21:07:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 21:07:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 21:15:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bbc59874-7510-aaf2-d7da-34ef94753588 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881486020400, cur 1539749740 expire 1539749590 last 1539749513 Oct 16 21:15:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 16 21:17:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 21:21:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 16 21:29:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 21:29:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 21:29:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e47397e-1d24-ed44-015a-603d4a9baad6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881250753400, cur 1539750599 expire 1539750449 last 1539750372 Oct 16 21:29:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 21:42:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 69e54a1a-fcfb-dbf7-dd3f-c2fd06b6d012 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f3ccff800, cur 1539751323 expire 1539751173 last 1539751096 Oct 16 21:42:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 21:56:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 16 21:56:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 22:26:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 22:29:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 22:29:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f370802d-f6a7-88a7-24a3-a12b14131d6d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d22800, cur 1539754190 expire 1539754040 last 1539753963 Oct 16 22:29:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 22:35:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9483e76a-b92a-6bf3-bbc7-c818d18dc3bb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fb5c5000, cur 1539754546 expire 1539754396 last 1539754319 Oct 16 22:39:56 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 22:43:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 22:46:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 22:47:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4cdd681c-269f-3194-31dd-d583a28c56d1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88203ab2e000, cur 1539755240 expire 1539755090 last 1539755013 Oct 16 22:51:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 22:56:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9cef229b-4fe1-2f11-47d4-96867ca3e572 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ce878e400, cur 1539755776 expire 1539755626 last 1539755549 Oct 16 22:56:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 22:57:07 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 22:57:14 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 22:57:14 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 22:58:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 23:02:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 23:03:04 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 23:05:16 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 23:05:16 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 16 23:09:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client af793d84-b5c6-bff5-53c0-760b70532600 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e17d9e000, cur 1539756563 expire 1539756413 last 1539756336 Oct 16 23:09:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 23:11:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 23:14:47 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 23:14:47 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 16 23:15:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 23:17:41 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 23:17:41 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 16 23:21:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 23:21:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3dea69db-5d69-279b-eed5-320094af346c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef00a400, cur 1539757296 expire 1539757146 last 1539757069 Oct 16 23:21:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 23:23:08 oak-md1-s2 kernel: Lustre: 10659:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 23:26:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 23:31:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7dbdb58a-7cd3-2a48-f758-3c7d22f30fae (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e0a6fbc00, cur 1539757918 expire 1539757768 last 1539757691 Oct 16 23:31:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 23:36:57 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 16 23:36:57 oak-md1-s2 kernel: Lustre: 10499:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 16 23:39:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 16 23:39:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 23:47:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05e38841-7b5f-db2c-b00e-dfe16e6a4f50 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe9378400, cur 1539758823 expire 1539758673 last 1539758596 Oct 16 23:47:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 16 23:50:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 00:02:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e2a55d0-532f-c3f7-4b2c-2a7c97d4982a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa1a8400, cur 1539759721 expire 1539759571 last 1539759494 Oct 17 00:02:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 00:04:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 00:14:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 528f39cf-d423-459c-b598-fe4326553e7a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031dc76c00, cur 1539760492 expire 1539760342 last 1539760265 Oct 17 00:14:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 00:18:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 00:18:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 00:25:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4d7d7756-5763-dc66-6518-ea267751e52c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7b15c00, cur 1539761136 expire 1539760986 last 1539760909 Oct 17 00:31:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 00:31:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 00:36:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c7bbe2da-378c-a227-f96a-573ad9121e01 (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b66a26000, cur 1539761766 expire 1539761616 last 1539761539 Oct 17 00:36:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 00:44:25 oak-md1-s2 kernel: Lustre: 10591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 00:44:26 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 00:45:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 00:53:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c385d237-b4cc-cce5-c13b-d7d0cd0c0f9c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e6622800, cur 1539762801 expire 1539762651 last 1539762574 Oct 17 00:53:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 00:56:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 00:56:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 01:05:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 795b7d86-f3ba-a8b3-0d90-8a8c05981455 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88095fa1e800, cur 1539763549 expire 1539763399 last 1539763322 Oct 17 01:05:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 01:10:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 01:10:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 01:20:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a046a25a-c7d0-b8b2-3e47-038cb37f6d39 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567bb000, cur 1539764423 expire 1539764273 last 1539764196 Oct 17 01:20:23 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 01:22:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 01:22:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 01:36:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 441419cb-ee58-8771-08bd-d63474d5896d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815e56fb800, cur 1539765365 expire 1539765215 last 1539765138 Oct 17 01:36:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 01:38:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 01:38:12 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 01:49:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 814ea29b-a9b4-d960-b169-e47fc1cbdcbb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185baef000, cur 1539766157 expire 1539766007 last 1539765930 Oct 17 01:49:17 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 01:50:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 01:50:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 02:03:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b2dc32fd-344c-eac0-2c13-36577f6884e1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881875ecec00, cur 1539767023 expire 1539766873 last 1539766796 Oct 17 02:03:43 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 02:06:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 02:06:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 02:17:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 02:25:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 227edeea-8d6a-1f0d-edb7-fa9b20f48826 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eaa678000, cur 1539768334 expire 1539768184 last 1539768107 Oct 17 02:25:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 02:31:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c0a67fd-7a14-d5d7-1297-e4f1af6884bd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1c0bec00, cur 1539768714 expire 1539768564 last 1539768487 Oct 17 02:33:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 02:33:14 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 02:40:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ec3e0c75-53a1-6e54-7e57-564b29fcc0af (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e70795000, cur 1539769221 expire 1539769071 last 1539768994 Oct 17 02:46:06 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 02:46:06 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 17 02:46:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bcc31410-c282-a65f-0850-00bfd4733ba9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202ca33000, cur 1539769589 expire 1539769439 last 1539769362 Oct 17 02:46:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 02:49:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 02:49:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 03:00:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f083d63-9cf8-2e7c-0112-41ea61c4ad08 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88197aa5cc00, cur 1539770409 expire 1539770259 last 1539770182 Oct 17 03:00:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 03:02:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 03:02:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 03:13:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 03:13:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 03:14:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae7206cc-83a7-6517-a1b6-670e06e0120c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88197aa5ac00, cur 1539771244 expire 1539771094 last 1539771017 Oct 17 03:14:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 03:21:03 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 03:21:03 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 17 03:24:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db87f198-b8ed-c308-dd9d-a1a4e2e40309 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88068aa29800, cur 1539771887 expire 1539771737 last 1539771660 Oct 17 03:24:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 03:25:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 03:25:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 03:25:13 oak-md1-s2 kernel: Lustre: 10564:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 03:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8906928d-58dd-2f45-4a49-20841992aa30 (at 10.8.9.10@o2ib6) reconnecting Oct 17 03:37:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 538c434d-dc1e-aed6-0f12-b0a48d1f80cc (at 10.12.4.50@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880562b4a400, cur 1539772675 expire 1539772525 last 1539772448 Oct 17 03:37:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 03:40:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 03:40:25 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 03:49:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 43a764eb-131b-b4db-94a1-bf616ad0eb6b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dfc5b5000, cur 1539773366 expire 1539773216 last 1539773139 Oct 17 03:49:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 03:56:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 17 03:56:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 04:12:15 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 04:12:15 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 17 04:42:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8be5a957-eff5-d829-4294-36548cc2630d (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185baed800, cur 1539776555 expire 1539776405 last 1539776328 Oct 17 04:42:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 04:43:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b1e3a6bd-438a-2cd6-9c8f-bdded1f7e828 (at 10.8.17.26@o2ib6) in 164 seconds. I think it's dead, and I am evicting it. exp ffff8811eab20400, cur 1539776631 expire 1539776481 last 1539776467 Oct 17 04:50:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 04:55:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 878f2ab0-b4bb-adc5-dfbc-b391b84fc914 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c1fb85c00, cur 1539777315 expire 1539777165 last 1539777088 Oct 17 04:55:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 04:59:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 05:15:11 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 05:15:11 oak-md1-s2 kernel: Lustre: 10567:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 17 05:15:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4f8ff9b5-3c9f-09af-6ef9-a813290ae6d7 (at 10.8.14.8@o2ib6) Oct 17 05:27:01 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 05:27:01 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 17 05:31:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c72fa751-856c-b5bc-632b-51e82a4ca548 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880965ab6800, cur 1539779464 expire 1539779314 last 1539779237 Oct 17 05:31:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 05:33:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 05:36:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 05:37:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 483399e2-749f-2125-800a-e0b7d85297dc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc1875000, cur 1539779833 expire 1539779683 last 1539779606 Oct 17 05:41:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38e33a74-5354-4259-326d-62fd6f719c92 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc7fa6800, cur 1539780083 expire 1539779933 last 1539779856 Oct 17 05:44:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 05:48:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 05:48:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 449248d5-d704-4546-02df-55e3f212637d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d6e211400, cur 1539780524 expire 1539780374 last 1539780297 Oct 17 05:51:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 05:51:11 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 05:54:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6d2794fd-8f17-59f2-8c49-ab59833cf493 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d1464b800, cur 1539780892 expire 1539780742 last 1539780665 Oct 17 05:54:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 05:55:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 05:58:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 06:05:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0bf66065-719f-bad1-7618-e0428bce8667 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812383b3000, cur 1539781514 expire 1539781364 last 1539781287 Oct 17 06:05:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 06:08:01 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 06:08:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 06:08:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 06:17:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ed5a1f88-f48c-a7dd-2ff5-da774c81fe40 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567bd000, cur 1539782252 expire 1539782102 last 1539782025 Oct 17 06:17:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 06:18:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 06:18:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 06:22:00 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 06:29:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 461caa33-1ed5-e817-925e-4e38b99cb0a8 (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8811eab26c00, cur 1539782962 expire 1539782812 last 1539782735 Oct 17 06:29:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 06:29:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 06:41:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd079738-aa59-974b-6d43-ce2eda838d9b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c7fa23c00, cur 1539783701 expire 1539783551 last 1539783474 Oct 17 06:41:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 06:43:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 06:43:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 06:44:41 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 06:49:27 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 06:53:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 34381e57-36ca-c00f-860b-de66460da1d5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bebfad800, cur 1539784389 expire 1539784239 last 1539784162 Oct 17 06:55:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 06:55:14 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 07:03:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a39848d5-364e-9836-e590-abb9b42d4ecf (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dc196e000, cur 1539785001 expire 1539784851 last 1539784774 Oct 17 07:11:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 07:11:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 07:14:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3454bb93-8042-dbbe-38df-1722e20d9691 (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031f760400, cur 1539785674 expire 1539785524 last 1539785447 Oct 17 07:14:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 07:17:28 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 07:17:29 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 07:25:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 07:25:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 07:43:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9aa0db99-8a26-0180-561a-d930b8bfbe02 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cb2a79400, cur 1539787407 expire 1539787257 last 1539787180 Oct 17 07:43:27 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 07:46:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 07:50:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97e9b0fc-4eb3-3bca-2b86-2b27b788fe2a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac96cc00, cur 1539787844 expire 1539787694 last 1539787617 Oct 17 07:53:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 07:54:54 oak-md1-s2 kernel: Lustre: 4300:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 07:58:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1d069625-8d2d-e5f7-8195-7247de8b2c5c (at 10.8.1.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b2b4cc00, cur 1539788288 expire 1539788138 last 1539788061 Oct 17 07:58:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 738ce072-f2f6-5243-bdb6-bd10f30bff16 (at 10.8.1.4@o2ib6) Oct 17 08:06:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 08:10:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c849c2d-ac3c-9539-ddeb-cf8a5eaf4665 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882005c36c00, cur 1539789005 expire 1539788855 last 1539788778 Oct 17 08:10:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 08:18:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 08:18:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 08:21:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d1df7513-5b96-089f-21f0-7f75d5463f5a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881be3367c00, cur 1539789677 expire 1539789527 last 1539789450 Oct 17 08:21:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 08:23:51 oak-md1-s2 kernel: Lustre: 4276:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 08:32:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5774023e-4c0d-7cdf-3b8d-3ff9c0d58d45 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88180feb3400, cur 1539790320 expire 1539790170 last 1539790093 Oct 17 08:33:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 08:33:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 08:43:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 13b8e08f-65d8-7088-7819-3396adbf1d47 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809dc2a6400, cur 1539790991 expire 1539790841 last 1539790764 Oct 17 08:43:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 08:45:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 08:45:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 08:48:00 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 08:48:00 oak-md1-s2 kernel: Lustre: 10521:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 17 08:54:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 82532717-63ee-ad81-23f6-5287cd69727f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881812763c00, cur 1539791690 expire 1539791540 last 1539791463 Oct 17 08:54:50 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 08:56:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 08:56:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 09:09:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 09:09:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 09:09:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd34af0a-1e92-82ee-8edb-ac968d9d5a30 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d6e212000, cur 1539792598 expire 1539792448 last 1539792371 Oct 17 09:09:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 09:21:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 09:21:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 09:28:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dde043ba-9b16-a2d4-56e0-2788b6fc026b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee7553800, cur 1539793686 expire 1539793536 last 1539793459 Oct 17 09:28:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 09:32:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 09:32:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 09:44:34 oak-md1-s2 kernel: Lustre: 10565:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539794667/real 1539794667] req@ffff88069ac9ad00 x1610633055386000/t0(0) o104->oak-MDT0000@10.8.18.32@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539794674 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 17 09:44:34 oak-md1-s2 kernel: Lustre: 10565:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 17 09:44:48 oak-md1-s2 kernel: Lustre: 10565:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1539794681/real 1539794681] req@ffff88069ac9ad00 x1610633055386000/t0(0) o104->oak-MDT0000@10.8.18.32@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1539794688 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 17 09:44:48 oak-md1-s2 kernel: Lustre: 10565:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 17 09:45:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cde93caf-a0f3-36e4-9140-691316d1f75f (at 10.8.18.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88059c298800, cur 1539794701 expire 1539794551 last 1539794474 Oct 17 09:45:01 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 09:47:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 09:47:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 09:56:30 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 09:56:30 oak-md1-s2 kernel: Lustre: 4882:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 17 09:58:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53d0f3da-ba84-99d7-1f60-4d42035c887b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef102000, cur 1539795514 expire 1539795364 last 1539795287 Oct 17 09:58:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 09:58:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 09:58:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 10:02:51 oak-md1-s2 kernel: Lustre: 10510:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:09:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 84b30ff3-488e-6243-89b7-d40f1aed2c6e (at 10.8.18.32@o2ib6) Oct 17 10:09:04 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 17 10:10:21 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:10:21 oak-md1-s2 kernel: Lustre: 10491:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 17 10:12:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5ade9ef-9505-2ee4-667c-6e38a7c3e2ba (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080dbc5000, cur 1539796340 expire 1539796190 last 1539796113 Oct 17 10:12:20 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 17 10:14:23 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:14:23 oak-md1-s2 kernel: Lustre: 10630:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 17 10:15:17 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:15:17 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 17 10:19:27 oak-md1-s2 kernel: Lustre: 10656:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:22:22 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:22:22 oak-md1-s2 kernel: Lustre: 10550:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 17 10:22:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 10:22:52 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 10:23:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ea09008-32ff-f20d-9aff-0cf1611a6aa6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102e3a0c00, cur 1539796998 expire 1539796848 last 1539796771 Oct 17 10:23:18 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 10:23:59 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:23:59 oak-md1-s2 kernel: Lustre: 10524:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 17 10:33:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 17 10:33:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 10:36:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f1d6eeea-96f0-9a65-8c02-21a4afc72bfd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814f06d4800, cur 1539797770 expire 1539797620 last 1539797543 Oct 17 10:36:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 10:40:33 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:40:33 oak-md1-s2 kernel: Lustre: 10606:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages Oct 17 10:48:12 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:48:12 oak-md1-s2 kernel: Lustre: 10553:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 17 10:52:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 68cf0a4e-f12b-1990-d9f2-59b4c1a9d8f1 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f593dc800, cur 1539798726 expire 1539798576 last 1539798499 Oct 17 10:52:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 17 10:52:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 10:57:00 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 10:57:00 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 17 11:04:09 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 11:04:09 oak-md1-s2 kernel: Lustre: 10542:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 17 11:08:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 31c3189e-adea-2227-7948-85cc244a18b9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f367000, cur 1539799690 expire 1539799540 last 1539799463 Oct 17 11:08:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 11:10:27 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 11:10:27 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 17 11:11:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 11:11:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 11:21:15 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 11:21:15 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 17 11:23:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 11:23:34 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 11:23:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f78e26f5-1459-be26-a8fd-8d7ddbc6966d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a94328800, cur 1539800624 expire 1539800474 last 1539800397 Oct 17 11:23:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 11:25:59 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 11:25:59 oak-md1-s2 kernel: Lustre: 10509:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 17 11:33:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e128f4b-5d75-fa9e-5e35-8b7a64463c73 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ffae96400, cur 1539801235 expire 1539801085 last 1539801008 Oct 17 11:33:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 11:40:20 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 11:40:20 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 17 11:41:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 11:41:05 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 11:45:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6a875080-12a1-8cea-cf28-d13939a4489a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804f20a3000, cur 1539801925 expire 1539801775 last 1539801698 Oct 17 11:45:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 11:52:57 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 11:52:57 oak-md1-s2 kernel: Lustre: 10561:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 17 11:54:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 11:54:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 11:59:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 44c1734f-e8cd-8d0c-a9b1-a4a81a2d223e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800585fac00, cur 1539802749 expire 1539802599 last 1539802522 Oct 17 11:59:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 12:05:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 12:05:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 12:08:40 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 12:08:40 oak-md1-s2 kernel: Lustre: 10641:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 17 12:10:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 433be208-2e10-7985-48a8-038bc93f1b70 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f710c00, cur 1539803406 expire 1539803256 last 1539803179 Oct 17 12:10:06 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 12:20:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 12:20:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 12:24:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53bce9e9-75a0-5c9f-ba90-e53ac6a736fb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818893efc00, cur 1539804257 expire 1539804107 last 1539804030 Oct 17 12:24:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 12:30:09 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 12:30:09 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 58 previous similar messages Oct 17 12:32:49 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 12:32:49 oak-md1-s2 kernel: Lustre: 10637:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 17 12:35:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0e62b68-3f51-d71f-8059-86cbb29bfc75 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8afc000, cur 1539804922 expire 1539804772 last 1539804695 Oct 17 12:35:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 12:36:57 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 12:36:57 oak-md1-s2 kernel: Lustre: 10531:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Oct 17 12:38:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 12:38:20 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 12:42:00 oak-md1-s2 kernel: Lustre: 10517:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 12:48:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c111cdc0-defc-01ab-0115-45b7178c60dd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a45b74800, cur 1539805680 expire 1539805530 last 1539805453 Oct 17 12:48:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 12:52:02 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 12:52:02 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 17 12:54:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 12:54:45 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 12:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a79f8ff-b0a6-cb59-c917-1d2a6c9223fb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88037c15e800, cur 1539806312 expire 1539806162 last 1539806085 Oct 17 12:58:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 13:02:27 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 13:02:27 oak-md1-s2 kernel: Lustre: 10574:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages Oct 17 13:13:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 13:13:06 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 13:13:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 334985f7-49c7-0c6f-9490-e8a1d4690613 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880761e33800, cur 1539807189 expire 1539807039 last 1539806962 Oct 17 13:13:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 13:13:31 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 13:13:31 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 44 previous similar messages Oct 17 13:28:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client de6e8803-8e21-3142-e404-8f052b469c49 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880be4fac400, cur 1539808121 expire 1539807971 last 1539807894 Oct 17 13:28:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 13:30:36 oak-md1-s2 kernel: Lustre: 10530:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 13:30:36 oak-md1-s2 kernel: Lustre: 10530:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 17 13:43:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 17 13:56:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 17 13:56:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51335e9b-01f1-9dc8-a240-2e0646378e5d (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b67284000, cur 1539809774 expire 1539809624 last 1539809547 Oct 17 13:57:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e16de0c5-aabc-5e26-52e3-a3938957ee22 (at 10.8.13.14@o2ib6) Oct 17 13:57:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 188d948b-39ee-df08-61d3-a715cbfc323e (at 10.8.17.26@o2ib6) in 163 seconds. I think it's dead, and I am evicting it. exp ffff8804da19dc00, cur 1539809850 expire 1539809700 last 1539809687 Oct 17 13:57:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 13:58:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 17 14:00:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 14:04:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e16de0c5-aabc-5e26-52e3-a3938957ee22 (at 10.8.13.14@o2ib6) Oct 17 14:04:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79089e01-8e26-f267-5222-52ddf7082a4e (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daaa39400, cur 1539810287 expire 1539810137 last 1539810060 Oct 17 14:04:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 14:09:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 14:09:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 14:09:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 731ea2ea-2559-4dc8-81b7-9ac61b81d82b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881293368400, cur 1539810589 expire 1539810439 last 1539810362 Oct 17 14:09:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 14:17:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 14:17:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 14:20:25 oak-md1-s2 kernel: Lustre: 10530:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 14:22:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bb0f4ec9-8e8d-c51e-820b-e0d9a085636c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc49000, cur 1539811378 expire 1539811228 last 1539811151 Oct 17 14:22:58 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 14:29:20 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 14:29:20 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 17 14:32:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 14:32:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 14:33:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 741a26eb-2cd2-ef3e-361f-b27d82942551 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88145768f400, cur 1539811999 expire 1539811849 last 1539811772 Oct 17 14:33:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 14:43:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e16de0c5-aabc-5e26-52e3-a3938957ee22 (at 10.8.13.14@o2ib6) Oct 17 14:43:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 14:44:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 89bd6aac-1db8-9322-b5b1-8e7c75ed4bb0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000ee76800, cur 1539812677 expire 1539812527 last 1539812450 Oct 17 14:44:37 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 14:54:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 14:54:59 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 14:55:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7de6e6b-27c0-a633-ef93-e299cb106ed5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881698e96800, cur 1539813314 expire 1539813164 last 1539813087 Oct 17 14:55:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 15:07:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 15:07:32 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 15:08:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 61bf5fba-e7b9-2685-caf7-ce8019c0fbd0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d050000, cur 1539814087 expire 1539813937 last 1539813860 Oct 17 15:08:07 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 15:21:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 17 15:21:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 15:21:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f1577c6-f8bf-b837-49ce-db5ac103e790 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103b5f0400, cur 1539814883 expire 1539814733 last 1539814656 Oct 17 15:21:23 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 15:32:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 12586e6d-0cf8-d521-d41a-2754ad7d760a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7b11c00, cur 1539815571 expire 1539815421 last 1539815344 Oct 17 15:32:51 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 15:34:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 15:34:20 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 15:37:13 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 15:37:13 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 17 15:42:04 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 15:42:04 oak-md1-s2 kernel: Lustre: 4322:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 17 15:44:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b6d3537-c7dd-8591-0428-cd8d43d7fab5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880185a68000, cur 1539816255 expire 1539816105 last 1539816028 Oct 17 15:44:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 15:49:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 15:49:26 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 17 15:49:33 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 15:55:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f1e8269e-31b8-2643-fe82-6e90dba9357c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daab08c00, cur 1539816919 expire 1539816769 last 1539816692 Oct 17 15:55:19 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 16:00:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 17 16:00:42 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 16:03:21 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 16:03:21 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 17 16:07:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 278c5307-fed9-2857-a8d4-3b955cafdbd5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040e0f400, cur 1539817626 expire 1539817476 last 1539817399 Oct 17 16:07:06 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 16:10:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 16:10:49 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 16:20:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c1c89682-dc21-f584-cf5d-d29dcc6b65cd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88006763e800, cur 1539818401 expire 1539818251 last 1539818174 Oct 17 16:20:01 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 17 16:22:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 17 16:22:22 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 16:33:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 16:33:59 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 16:34:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5dadaa92-de89-3e00-9593-097a0886c9ed (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef009800, cur 1539819281 expire 1539819131 last 1539819054 Oct 17 16:34:41 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 17 16:47:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 16:47:53 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 16:49:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c333ffb-b40c-1fa1-653c-6033c7c1533b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880057719c00, cur 1539820142 expire 1539819992 last 1539819915 Oct 17 16:49:02 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 17 16:52:31 oak-md1-s2 kernel: Lustre: 4294:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 16:59:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 16:59:14 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 17:03:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04ddd0dd-5cf8-3e6b-f85b-b62cd06aab56 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f593d9800, cur 1539820980 expire 1539820830 last 1539820753 Oct 17 17:03:00 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 17:04:13 oak-md1-s2 kernel: Lustre: 10676:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 17:20:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 17ce4421-a549-5aa5-9a17-c7c3e1ec62ab (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe534ac00, cur 1539822006 expire 1539821856 last 1539821779 Oct 17 17:20:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 17:21:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 17:21:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 17:25:11 oak-md1-s2 kernel: Lustre: 10485:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 17:25:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 17:28:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 17:30:10 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 17:35:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 17:36:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eb7d8db4-3d5c-9d73-e351-2a1baff717d3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014e25d000, cur 1539822986 expire 1539822836 last 1539822759 Oct 17 17:36:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 17:49:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 17:49:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 17:50:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f14404f-1f64-3de7-4b17-bd35ce9eceb8 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053efac00, cur 1539823817 expire 1539823667 last 1539823590 Oct 17 17:50:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 18:02:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e6e562d-c4ce-081c-135b-b022fcf90467 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0e7000, cur 1539824534 expire 1539824384 last 1539824307 Oct 17 18:02:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 18:02:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 18:02:23 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 17 18:15:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6663eda9-84e1-5508-801c-be9ab7360f97 (at 10.12.4.35@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ddcc4b800, cur 1539825323 expire 1539825173 last 1539825096 Oct 17 18:15:23 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 18:36:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 124fd82b-64d3-006d-30bd-4491beb16bac (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e4ac26000, cur 1539826601 expire 1539826451 last 1539826374 Oct 17 18:38:56 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 18:48:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 17 18:48:09 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 17 18:49:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 368bfd90-06f0-5aed-002c-357cfa619f24 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daca13800, cur 1539827343 expire 1539827193 last 1539827116 Oct 17 18:52:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3d91a8e7-83a7-6721-796e-b49d4dbc7b20 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dab42f000, cur 1539827579 expire 1539827429 last 1539827352 Oct 17 19:04:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 17 19:04:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 19:12:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 19:13:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a048184-64ce-b2d9-053c-7e68297a4ca2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003a19f400, cur 1539828789 expire 1539828639 last 1539828562 Oct 17 19:23:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc42f025-ed77-fef4-8f7b-53adde06220b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf27dc00, cur 1539829410 expire 1539829260 last 1539829183 Oct 17 19:25:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 19:30:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 19:30:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 81d121a5-80fd-f0fd-be34-7ab11433dc31 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881856ff0c00, cur 1539829834 expire 1539829684 last 1539829607 Oct 17 19:47:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e54fc1a7-2fde-b72d-e225-d35660488d0f (at 10.12.4.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ef8785000, cur 1539830858 expire 1539830708 last 1539830631 Oct 17 19:48:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 08b6f6ce-240a-77f6-05f3-c45b70f741fa (at 10.12.4.50@o2ib) Oct 17 19:48:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa5541c6-b49f-97f9-7ebd-59eaa5151787 (at 10.12.4.35@o2ib) Oct 17 19:48:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0c7eefa3-fc99-540c-06e7-6b75b4a7596a (at 10.12.4.71@o2ib) Oct 17 19:48:54 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 20:02:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8173294d-1739-99a3-a491-3d82d2b49615 (at 10.12.4.35@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272377c00, cur 1539831742 expire 1539831592 last 1539831515 Oct 17 20:13:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75a51f67-a28b-bf6d-1b07-8411b656cff8 (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88048a9ed800, cur 1539832436 expire 1539832286 last 1539832209 Oct 17 20:15:51 oak-md1-s2 kernel: LustreError: 10556:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.49@o2ib4) failed to reply to blocking AST (req@ffff8803b1a1e600 x1610633223232096 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8808b0e46000/0x6c00c41b792bf65c lrc: 5/0,0 mode: PR/PR res: [0x200017ac0:0x1b551:0x0].0x0 bits 0x1b rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.9.105.49@o2ib4 remote: 0xe790423c3574138f expref: 3584 pid: 10556 timeout: 0 lvb_type: 0 Oct 17 20:15:51 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.49@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Oct 17 21:02:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 086bc54c-a5cd-8fa6-98e1-6354f6af0134 (at 10.12.4.50@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0e6800, cur 1539835339 expire 1539835189 last 1539835112 Oct 17 21:10:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7073ab44-48a9-1d3f-564a-bce89ffc386b (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93136000, cur 1539835821 expire 1539835671 last 1539835594 Oct 17 21:11:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 21:16:27 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 21:31:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c750ac1-e6b5-8118-8390-4cd65b4f547f (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea84dac00, cur 1539837095 expire 1539836945 last 1539836868 Oct 17 21:50:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f391cb18-2dea-a403-02b9-7c06005e60c4 (at 10.9.102.25@o2ib4) Oct 17 21:51:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4fb46c22-f0cb-3044-53f0-d95dd03a4244 (at 10.9.102.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cb2a79c00, cur 1539838291 expire 1539838141 last 1539838064 Oct 17 21:53:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 22:09:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 08b6f6ce-240a-77f6-05f3-c45b70f741fa (at 10.12.4.50@o2ib) Oct 17 22:16:27 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 22:16:51 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 22:16:51 oak-md1-s2 kernel: Lustre: 10504:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 17 22:22:29 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 17 22:22:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 38ee1e02-320f-be1c-55df-c055eb6f7b2f (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e20d7ec00, cur 1539840179 expire 1539840029 last 1539839952 Oct 17 22:24:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5bff813e-bb10-564e-85b8-f2b39b8c6308 (at 10.8.17.26@o2ib6) in 190 seconds. I think it's dead, and I am evicting it. exp ffff88103d67cc00, cur 1539840255 expire 1539840105 last 1539840065 Oct 17 22:25:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 22:29:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 496d3087-dd9b-77b5-6203-93cbe501a187 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db0210400, cur 1539840569 expire 1539840419 last 1539840342 Oct 17 22:31:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 22:35:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7652f783-8941-addc-94eb-408d5991b442 (at 10.8.10.6@o2ib6) Oct 17 22:36:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 46834d38-9ed2-004f-4ef7-92a64dc9e623 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88149524c800, cur 1539840967 expire 1539840817 last 1539840740 Oct 17 22:37:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 22:39:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 45061aa3-dc1e-a1b7-c893-02891f2af7d7 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d4b596400, cur 1539841163 expire 1539841013 last 1539840936 Oct 17 22:41:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 22:41:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a6312ee-d404-3f1d-4281-4aacadefc314 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880041c1d800, cur 1539841319 expire 1539841169 last 1539841092 Oct 17 22:45:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c4045a9b-0fca-68bf-541d-01f7ae8e41b4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa3f2800, cur 1539841553 expire 1539841403 last 1539841326 Oct 17 22:49:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 22:54:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 22:55:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 575271b7-43af-6417-179b-219dc7af9cee (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816102c5400, cur 1539842112 expire 1539841962 last 1539841885 Oct 17 22:58:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 22:58:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2db5cbf2-9f59-9f51-5589-b6af243cbe0d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb91c00, cur 1539842319 expire 1539842169 last 1539842092 Oct 17 23:02:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4ec2ae9-7029-f8b5-303d-0ea1a757e80a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb2b000, cur 1539842578 expire 1539842428 last 1539842351 Oct 17 23:05:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 23:09:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9afcb07d-1195-8aca-f42a-2d7351c732dd (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25731000, cur 1539842948 expire 1539842798 last 1539842721 Oct 17 23:11:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 23:18:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 23:18:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 23:18:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3a00a9e5-4826-35da-3a0e-6af25e3ca121 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea84d9800, cur 1539843522 expire 1539843372 last 1539843295 Oct 17 23:18:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 23:28:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b5d83faf-c733-9ee1-bfaa-17269de44e8a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b2689800, cur 1539844132 expire 1539843982 last 1539843905 Oct 17 23:28:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 23:31:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 23:31:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 23:41:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 23:47:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b9a0d866-7369-72d9-622f-e232b29643ae (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801530ebc00, cur 1539845267 expire 1539845117 last 1539845040 Oct 17 23:47:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 23:52:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 17 23:52:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 17 23:56:18 oak-md1-s2 kernel: Lustre: 10556:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 00:01:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 78727892-0634-d278-1358-b512b7139f58 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe9df0000, cur 1539846091 expire 1539845941 last 1539845864 Oct 18 00:01:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 00:07:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 00:07:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 00:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65c3a49e-8cea-6a3b-d798-5f1d7f11f26f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e4ac25400, cur 1539846803 expire 1539846653 last 1539846576 Oct 18 00:13:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 00:18:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 00:18:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 00:28:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9eec994c-fd1e-1f69-802a-e7dfb8d66aa9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880185a6dc00, cur 1539847712 expire 1539847562 last 1539847485 Oct 18 00:28:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 00:30:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 00:30:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 00:41:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 00:41:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 00:42:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 591f1468-d770-1d1c-899d-6c39ae22d330 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a66cd8000, cur 1539848545 expire 1539848395 last 1539848318 Oct 18 00:42:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 00:55:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 00:55:44 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 00:56:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1c2f1098-1d71-9871-2a4a-ba71baa47814 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bdeb8b400, cur 1539849367 expire 1539849217 last 1539849140 Oct 18 00:56:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 01:06:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 01:06:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 01:10:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7d6ef94d-a9ee-e43d-646f-8ff9a6c63fc7 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88102cfd8800, cur 1539850224 expire 1539850074 last 1539849997 Oct 18 01:10:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 01:19:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 01:19:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 01:23:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8d906461-d77a-2800-bd5d-b01aa20061fc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184b686000, cur 1539851031 expire 1539850881 last 1539850804 Oct 18 01:23:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 01:32:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 01:32:21 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 01:34:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd6f95a8-6b71-f665-de84-729ec83f3002 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e8b8cc000, cur 1539851693 expire 1539851543 last 1539851466 Oct 18 01:34:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 01:45:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 01:45:18 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 18 01:45:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 467d500d-fbd1-9c60-11b1-099ed8229b05 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11864000, cur 1539852324 expire 1539852174 last 1539852097 Oct 18 01:45:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 02:00:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 02:00:29 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 02:00:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4c29bc33-c34c-98c7-dd8d-4ec3fc4cce7c (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88029d91bc00, cur 1539853253 expire 1539853103 last 1539853026 Oct 18 02:00:53 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 02:31:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 45ff7f74-203b-b030-bf26-490736e5cefc (at 10.8.2.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801530efc00, cur 1539855095 expire 1539854945 last 1539854868 Oct 18 05:27:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7300a7c6-a65e-7368-5057-c10beec08cc4 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacde9c00, cur 1539865622 expire 1539865472 last 1539865395 Oct 18 05:27:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 18 05:28:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fecacfc4-7441-e228-fa5b-34ce99980324 (at 10.8.17.26@o2ib6) in 216 seconds. I think it's dead, and I am evicting it. exp ffff881f2b724800, cur 1539865698 expire 1539865548 last 1539865482 Oct 18 05:31:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 05:31:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 05:32:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a41e21d5-19c1-2df0-137e-9f4023b762ce (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031f760400, cur 1539865949 expire 1539865799 last 1539865722 Oct 18 05:36:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1c53568e-27f2-f33e-9500-125c3c7bac7a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880002afec00, cur 1539866167 expire 1539866017 last 1539865940 Oct 18 05:36:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 05:40:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf8afb71-35dd-62b7-a212-a44e3b9a2913 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c3aea7400, cur 1539866401 expire 1539866251 last 1539866174 Oct 18 05:43:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 05:43:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 05:43:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c774286a-e598-d78f-d3eb-d2dfd5cfddce (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88077a781800, cur 1539866629 expire 1539866479 last 1539866402 Oct 18 05:53:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b54da8eb-0baa-3ffb-55de-17f70500eedc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099725ec00, cur 1539867209 expire 1539867059 last 1539866982 Oct 18 05:53:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 05:56:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 05:56:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 06:06:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 06:06:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:06:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51180606-c01b-8f98-13b0-60bf0a886eb4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880adae73c00, cur 1539868019 expire 1539867869 last 1539867792 Oct 18 06:06:59 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:16:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 06:16:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:17:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ba9823fb-836f-73a4-0c73-7b57fff48c09 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880877cea400, cur 1539868653 expire 1539868503 last 1539868426 Oct 18 06:17:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:27:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 06:27:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:28:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe46bb92-403d-05d9-89f7-8e785f3f6cce (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880392e3f400, cur 1539869289 expire 1539869139 last 1539869062 Oct 18 06:28:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:39:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 06:39:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:39:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 195063be-760f-2fd5-62a2-f91a4a485da5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881246ae4400, cur 1539869991 expire 1539869841 last 1539869764 Oct 18 06:39:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 06:52:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fbaecae3-8f28-b637-d8a7-2c2057eee158 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880050f9ec00, cur 1539870748 expire 1539870598 last 1539870521 Oct 18 06:55:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 07:11:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c099268-27d4-fe5c-cc9c-8748c58e40ff (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807659ed400, cur 1539871905 expire 1539871755 last 1539871678 Oct 18 07:11:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 07:25:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 07:25:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 07:25:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 56709199-95a8-b9ac-5a7a-880c4f5aad41 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db00d1c00, cur 1539872758 expire 1539872608 last 1539872531 Oct 18 07:25:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 07:36:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 07:36:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 07:36:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4961d2b0-a783-6bda-a9e5-a47833f5b5b1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88183ae95c00, cur 1539873376 expire 1539873226 last 1539873149 Oct 18 07:36:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 07:47:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 07:47:29 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 07:48:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f780e37b-5c29-cd86-77f2-99941f0d2909 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880dfb596400, cur 1539874089 expire 1539873939 last 1539873862 Oct 18 07:48:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 07:59:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 07:59:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 07:59:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bc6e8e97-0813-149d-334a-c0b636fbd640 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff78f2400, cur 1539874794 expire 1539874644 last 1539874567 Oct 18 07:59:54 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:13:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 08:13:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:14:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be7f986e-805b-969f-2c73-5a1fa5dc48af (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880530fdc000, cur 1539875673 expire 1539875523 last 1539875446 Oct 18 08:14:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:26:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 08:26:49 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:27:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6683502d-84f8-8b70-1237-aa0aae58227f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88183ae97800, cur 1539876444 expire 1539876294 last 1539876217 Oct 18 08:27:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:38:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 08:38:24 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:39:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe985b38-953d-05ec-1826-08335c8fc012 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf070400, cur 1539877140 expire 1539876990 last 1539876913 Oct 18 08:39:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:52:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 08:52:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 08:53:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ab081371-16f9-6b70-38aa-1957f47beb19 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e785800, cur 1539877990 expire 1539877840 last 1539877763 Oct 18 08:53:10 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:07:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 09:07:49 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:07:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79684685-a555-2040-1025-c5301aabfff6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88041fb3e400, cur 1539878879 expire 1539878729 last 1539878652 Oct 18 09:07:59 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:20:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 09:20:04 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:20:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1bb5c2bc-89e8-e31f-0def-7a8f9b1d8c7b (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88184eb1a800, cur 1539879656 expire 1539879506 last 1539879429 Oct 18 09:20:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:33:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d9596860-9b88-9685-b0e0-f7e9cdbc0a05 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bf6579000, cur 1539880398 expire 1539880248 last 1539880171 Oct 18 09:33:18 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:33:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 09:33:18 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:46:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 09:46:40 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:47:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 865f8a13-d7b8-243a-c9d1-db59dbbc3beb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880050f9b000, cur 1539881235 expire 1539881085 last 1539881008 Oct 18 09:47:15 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 09:59:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 09:59:32 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 10:00:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 560bf6a0-8666-6213-4e7d-8b99d9b4aa7d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cf4768c00, cur 1539882012 expire 1539881862 last 1539881785 Oct 18 10:00:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 10:05:21 oak-md1-s2 kernel: Lustre: 3869:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:10:33 oak-md1-s2 kernel: Lustre: 10576:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:11:11 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:11:11 oak-md1-s2 kernel: Lustre: 10640:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 18 10:14:47 oak-md1-s2 kernel: Lustre: 10506:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:21:17 oak-md1-s2 kernel: Lustre: 10483:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:28:44 oak-md1-s2 kernel: Lustre: 10633:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:31:44 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:31:44 oak-md1-s2 kernel: Lustre: 10638:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 18 10:33:31 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:33:31 oak-md1-s2 kernel: Lustre: 10605:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 18 10:44:32 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:44:32 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 18 10:47:51 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:47:51 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 18 10:54:43 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 10:54:43 oak-md1-s2 kernel: Lustre: 10511:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 18 11:04:56 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 11:04:56 oak-md1-s2 kernel: Lustre: 10580:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages Oct 18 11:15:11 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 11:15:11 oak-md1-s2 kernel: Lustre: 10505:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 18 11:27:45 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 11:27:45 oak-md1-s2 kernel: Lustre: 10612:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 18 11:45:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to oak-MDT0001-mdtlov_UUID (at 10.0.2.51@o2ib5) Oct 18 11:45:30 oak-md1-s2 kernel: Lustre: ctl-oak-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:ost Oct 18 11:45:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 11:48:05 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 11:48:05 oak-md1-s2 kernel: Lustre: 10547:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 30 previous similar messages Oct 18 11:51:57 oak-md1-s2 kernel: Lustre: 10492:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 11:57:27 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 11:57:27 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 18 12:09:45 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 12:09:45 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 18 12:30:21 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 12:30:21 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 18 13:01:13 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 13:01:13 oak-md1-s2 kernel: Lustre: 10558:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 18 14:57:49 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 14:57:49 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 18 15:14:06 oak-md1-s2 kernel: Lustre: 10551:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:25:18 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:25:30 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:25:30 oak-md1-s2 kernel: Lustre: 10548:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 34 previous similar messages Oct 18 15:33:43 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:33:43 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 18 15:35:09 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:35:09 oak-md1-s2 kernel: Lustre: 10654:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 18 15:41:10 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:41:10 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages Oct 18 15:44:13 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:44:13 oak-md1-s2 kernel: Lustre: 10590:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages Oct 18 15:49:01 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 18 15:49:01 oak-md1-s2 kernel: Lustre: 4295:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 18 15:56:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1c46e295-6ebd-9028-7ab4-e9fde86a410f (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818292a9c00, cur 1539903363 expire 1539903213 last 1539903136 Oct 18 15:56:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 15:56:21 oak-md1-s2 kernel: LustreError: 4311:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881a5a387b00 x1610633437682928/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:56:21 oak-md1-s2 kernel: LustreError: 4311:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Oct 18 15:56:24 oak-md1-s2 kernel: LustreError: 4239:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88152ba63600 x1610633437691280/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:56:24 oak-md1-s2 kernel: LustreError: 4239:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Oct 18 15:56:30 oak-md1-s2 kernel: LustreError: 10571:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8813c272d700 x1610633437708240/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:56:37 oak-md1-s2 kernel: LustreError: 10537:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881ac0f72700 x1610633437734800/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:56:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 18 15:56:43 oak-md1-s2 kernel: LustreError: 10586:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881346ef1b00 x1610633437760112/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:56:43 oak-md1-s2 kernel: LustreError: 10586:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Oct 18 15:56:52 oak-md1-s2 kernel: LustreError: 4676:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881b7dbf0900 x1610633437817008/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:56:52 oak-md1-s2 kernel: LustreError: 4676:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Oct 18 15:57:08 oak-md1-s2 kernel: LustreError: 10586:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8816c9012d00 x1610633437913056/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:57:08 oak-md1-s2 kernel: LustreError: 10586:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Oct 18 15:57:52 oak-md1-s2 kernel: LustreError: 10506:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880369aad400 x1610633438063952/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:57:52 oak-md1-s2 kernel: LustreError: 10506:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 6 previous similar messages Oct 18 15:58:02 oak-md1-s2 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 10.0.2.3@o2ib5 ns: mdt-oak-MDT0000_UUID lock: ffff88057383a000/0x6c00c41ccc4ec70b lrc: 3/0,0 mode: PR/PR res: [0x200017bf3:0x6108:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.0.2.3@o2ib5 remote: 0xa03baa5799f06d82 expref: 752012 pid: 4694 timeout: 8187674765 lvb_type: 0 Oct 18 15:58:18 oak-md1-s2 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 10.0.2.3@o2ib5 ns: mdt-oak-MDT0000_UUID lock: ffff880d15386a00/0x6c00c41ccc55bd3a lrc: 3/0,0 mode: PR/PR res: [0x200017bf3:0x610d:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.0.2.3@o2ib5 remote: 0xa03baa5799f0b63e expref: 645207 pid: 4694 timeout: 8187690611 lvb_type: 0 Oct 18 15:58:21 oak-md1-s2 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 10.0.2.3@o2ib5 ns: mdt-oak-MDT0000_UUID lock: ffff880a879e2e00/0x6c00c41cc8d81eb1 lrc: 3/0,0 mode: PR/PR res: [0x200017bde:0x1ae76:0x0].0x0 bits 0x13 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.0.2.3@o2ib5 remote: 0xa03baa5799b2dfe8 expref: 626229 pid: 10548 timeout: 8187693104 lvb_type: 0 Oct 18 15:58:35 oak-md1-s2 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 10.0.2.3@o2ib5 ns: mdt-oak-MDT0000_UUID lock: ffff8802e5890000/0x6c00c41cc4a7eb1a lrc: 3/0,0 mode: PR/PR res: [0x200017e20:0xe292:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.0.2.3@o2ib5 remote: 0xa03baa5799474f85 expref: 535168 pid: 10644 timeout: 8187707685 lvb_type: 0 Oct 18 15:58:57 oak-md1-s2 kernel: LustreError: 10668:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881780008f00 x1610633438214944/t0(0) o104->oak-MDT0000@10.0.2.3@o2ib5:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 18 15:58:57 oak-md1-s2 kernel: LustreError: 10668:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 10 previous similar messages Oct 18 15:58:59 oak-md1-s2 kernel: LustreError: 0:0:(ldlm_lockd.c:334:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 10.0.2.3@o2ib5 ns: mdt-oak-MDT0000_UUID lock: ffff880e8a38f000/0x6c00c41ccc637f7c lrc: 3/0,0 mode: PR/PR res: [0x200017a2b:0x5d39:0x0].0x0 bits 0x1b rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.0.2.3@o2ib5 remote: 0xa03baa5799f14f50 expref: 392335 pid: 10567 timeout: 8187731914 lvb_type: 0 Oct 18 16:00:00 oak-md1-s2 kernel: LNet: Service thread pid 10579 was inactive for 200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Oct 18 16:00:00 oak-md1-s2 kernel: Pid: 10579, comm: mdt00_073 Oct 18 16:00:00 oak-md1-s2 kernel: Oct 18 16:00:00 oak-md1-s2 kernel: [] schedule+0x29/0x70 Oct 18 16:00:00 oak-md1-s2 kernel: [] schedule_timeout+0x174/0x2c0 Oct 18 16:00:00 oak-md1-s2 kernel: [] ? process_timeout+0x0/0x10 Oct 18 16:00:00 oak-md1-s2 kernel: [] ? cfs_block_sigsinv+0x71/0xa0 [libcfs] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] Oct 18 16:00:00 oak-md1-s2 kernel: [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? default_wake_function+0x0/0x20 Oct 18 16:00:00 oak-md1-s2 kernel: [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_object_local_lock+0x452/0xaf0 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? mdt_blocking_ast+0x0/0x2e0 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? lu_object_find_at+0x95/0x290 [obdclass] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_object_lock_internal+0x5e/0x300 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_object_lock+0x14/0x20 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_reint_open+0xc71/0x3190 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? ucred_set_jobid+0x53/0x70 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_reint_rec+0x80/0x210 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_reint_internal+0x5fb/0x9c0 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_intent_reint+0x162/0x430 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] mdt_intent_policy+0x43e/0xc70 [mdt] Oct 18 16:00:00 oak-md1-s2 kernel: [] ? ldlm_resource_get+0x9f/0xa30 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] ldlm_lock_enqueue+0x387/0x970 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] ldlm_handle_enqueue0+0x9c3/0x1680 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] tgt_enqueue+0x62/0x210 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] tgt_request_handle+0x925/0x1370 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] ? default_wake_function+0x12/0x20 Oct 18 16:00:01 oak-md1-s2 kernel: [] ? __wake_up_common+0x58/0x90 Oct 18 16:00:01 oak-md1-s2 kernel: [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Oct 18 16:00:01 oak-md1-s2 kernel: [] kthread+0xcf/0xe0 Oct 18 16:00:01 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Oct 18 16:00:01 oak-md1-s2 kernel: [] ret_from_fork+0x58/0x90 Oct 18 16:00:01 oak-md1-s2 kernel: [] ? kthread+0x0/0xe0 Oct 18 16:00:01 oak-md1-s2 kernel: Oct 18 16:00:01 oak-md1-s2 kernel: LustreError: dumping log to /tmp/lustre-log.1539903601.10579 Oct 18 16:00:10 oak-md1-s2 kernel: LNet: Service thread pid 10579 completed after 210.28s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Oct 18 16:00:10 oak-md1-s2 kernel: LNet: Skipped 3 previous similar messages Oct 18 16:09:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 18 16:10:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 18 16:10:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 18 16:51:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 18 16:51:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 18 17:16:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 18 17:16:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 18 18:05:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 18 18:05:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 719a16a6-f695-82bb-50cf-14a0498e6395 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803cb630800, cur 1539911140 expire 1539910990 last 1539910913 Oct 18 18:44:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 18 18:44:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 18 18:44:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a3b5e0bb-6507-771f-fc78-2c5e7eb536e1 (at 10.8.12.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bfa328000, cur 1539913490 expire 1539913340 last 1539913263 Oct 18 18:45:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 53462e99-45fa-ed22-9be8-74feb94dacbf (at 10.8.12.6@o2ib6) Oct 18 19:26:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 18 19:27:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e26f1122-dc6f-16a6-84ec-8e7222ec7b4e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88189aa8a400, cur 1539916062 expire 1539915912 last 1539915835 Oct 18 19:54:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 18 19:55:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 33a7dda3-2c0f-ab42-d473-474ed30a4930 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881856ff2400, cur 1539917712 expire 1539917562 last 1539917485 Oct 18 19:57:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 18 19:58:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9425e715-114c-1bdb-9788-a36472a28cb1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882001d9dc00, cur 1539917900 expire 1539917750 last 1539917673 Oct 18 19:58:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 20:54:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7f2b6f8-f5be-c86a-6c17-37fa9000507e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880970fffc00, cur 1539921265 expire 1539921115 last 1539921038 Oct 18 20:56:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:09:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bfc795c6-1bd3-2ed4-29d5-0b86070cab49 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c7b1400, cur 1539922186 expire 1539922036 last 1539921959 Oct 18 21:24:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:28:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 209be260-09b2-6aef-cb97-e280ae0263f7 (at 10.12.4.86@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880965ab4400, cur 1539923291 expire 1539923141 last 1539923064 Oct 18 21:28:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 44de14e0-2044-310b-52a4-6dee9193cd27 (at 10.12.4.85@o2ib) Oct 18 21:28:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d02addfd-1d23-3ae8-8dc1-18aa5d4cca28 (at 10.12.4.86@o2ib) Oct 18 21:29:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9fad85bb-7223-bbad-a44e-6ba1912e5f4c (at 10.12.4.87@o2ib) Oct 18 21:31:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:36:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f45d2cee-3f32-cbdb-c0b0-cd8a0bbd2a32 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88115e90d800, cur 1539923779 expire 1539923629 last 1539923552 Oct 18 21:36:19 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 18 21:36:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:40:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:40:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 28e63f1d-4866-c7ed-089d-9e0b291bdbad (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808552a3000, cur 1539924044 expire 1539923894 last 1539923817 Oct 18 21:46:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e33a6e7c-ae5a-6839-2c0b-7c5978e51050 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881befb81400, cur 1539924402 expire 1539924252 last 1539924175 Oct 18 21:48:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:54:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1f5ca4d5-4a45-0888-e95a-94a54d86da6e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88130aae2400, cur 1539924860 expire 1539924710 last 1539924633 Oct 18 21:54:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:54:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 18 21:58:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 21:58:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 21:59:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3af69cc8-8e76-371c-1da7-b8a0a0dcb47e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881540792000, cur 1539925151 expire 1539925001 last 1539924924 Oct 18 22:03:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 10e12056-0394-db23-2816-8d29ce9597b3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881002613c00, cur 1539925390 expire 1539925240 last 1539925163 Oct 18 22:06:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 22:10:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a34460b9-316a-3839-e105-83071435857f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ea0011c00, cur 1539925823 expire 1539925673 last 1539925596 Oct 18 22:17:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 22:17:49 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 22:18:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 53f0cd86-be8c-be40-c6f6-58b135d90b29 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf36d6c00, cur 1539926302 expire 1539926152 last 1539926075 Oct 18 22:21:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2156a10e-10a2-5643-5a04-16dd801cd739 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096bf07800, cur 1539926496 expire 1539926346 last 1539926269 Oct 18 22:25:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bec04fa4-6e73-1426-d258-2efb2c9fd006 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b3a3ec00, cur 1539926753 expire 1539926603 last 1539926526 Oct 18 22:28:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 22:28:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 22:34:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 18 22:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5fb34963-b70e-51d6-30fa-40dc549cf1a6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880353c3e800, cur 1539927287 expire 1539927137 last 1539927060 Oct 18 22:44:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f84ecc7-7057-ee12-249a-c73214e01cff (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bbf1dd000, cur 1539927892 expire 1539927742 last 1539927665 Oct 18 22:47:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 22:47:17 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 22:55:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9a390be-60cc-957a-3b8e-b911c0921f08 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039235c00, cur 1539928517 expire 1539928367 last 1539928290 Oct 18 22:55:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 22:57:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 22:57:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 23:15:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 18 23:15:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 18 23:15:57 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 23:26:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4fb0778a-9a56-9ffa-b92a-39eba1544d77 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb96000, cur 1539930395 expire 1539930245 last 1539930168 Oct 18 23:26:35 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 18 23:31:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 23:31:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 23:34:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c1432e7-51ab-aa4a-de88-81e0baf3e427 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88148eef5000, cur 1539930887 expire 1539930737 last 1539930660 Oct 18 23:38:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 925a9efe-2cb9-3e95-896f-a5452a70eb65 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88166b2e8c00, cur 1539931092 expire 1539930942 last 1539930865 Oct 18 23:42:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 23:42:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 18 23:46:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cee12c35-018c-f726-a0a4-fbc4197e73c5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851231800, cur 1539931572 expire 1539931422 last 1539931345 Oct 18 23:56:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 18 23:56:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 00:00:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be348272-1765-24c9-2562-e95a91e1cdda (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881832a61400, cur 1539932407 expire 1539932257 last 1539932180 Oct 19 00:00:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 00:10:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bbfa05db-12d9-bc7b-60ec-6bac74936b38 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88055c14e000, cur 1539933031 expire 1539932881 last 1539932804 Oct 19 00:15:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 00:15:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 00:24:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2945dc1-a481-76d5-d265-880f770343ab (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bfa329000, cur 1539933861 expire 1539933711 last 1539933634 Oct 19 00:24:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 00:28:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 00:28:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 00:39:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b1bdd51d-85b5-1329-3c9f-9fe94b0d83e0 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880814647800, cur 1539934764 expire 1539934614 last 1539934537 Oct 19 00:39:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 00:42:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 00:50:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea7f39f1-08fa-5fa0-020c-f5b8e784e6c1 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a30d400, cur 1539935448 expire 1539935298 last 1539935221 Oct 19 00:55:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 01:04:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bcf230f4-8779-c187-822b-286404352f69 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814f0362800, cur 1539936292 expire 1539936142 last 1539936065 Oct 19 01:04:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 01:12:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 01:12:06 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 01:17:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e912f346-548c-17e8-da20-8a9bc31a9372 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88175f954000, cur 1539937054 expire 1539936904 last 1539936827 Oct 19 01:17:34 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 01:22:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 01:22:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 01:28:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 72d0b158-b4e4-2e7d-b006-825c923bc3e2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804cb263400, cur 1539937711 expire 1539937561 last 1539937484 Oct 19 01:28:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 01:32:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 01:32:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 01:43:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9183aa4-656b-3c71-22d3-f6d2a92055eb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d32eec400, cur 1539938603 expire 1539938453 last 1539938376 Oct 19 01:43:23 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 01:48:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 01:48:05 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 01:54:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 68b8f304-03ad-4372-e841-35f26bcd8772 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880819d35000, cur 1539939263 expire 1539939113 last 1539939036 Oct 19 01:54:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 01:59:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 01:59:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 02:05:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e5ff673-94e2-dd8d-e43f-82a0aa7ea05e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d9a95d800, cur 1539939951 expire 1539939801 last 1539939724 Oct 19 02:05:51 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 02:27:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 19 02:27:31 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 02:28:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0fc56545-f495-3fa2-b4dd-41132940d014 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881875ec9c00, cur 1539941291 expire 1539941141 last 1539941064 Oct 19 02:30:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 19 02:31:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 143c77c0-5263-0496-0498-aa34886b95fe (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851233c00, cur 1539941512 expire 1539941362 last 1539941285 Oct 19 02:59:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e01ad40e-7b0c-ec6b-0da6-03131eef4833 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db0380400, cur 1539943172 expire 1539943022 last 1539942945 Oct 19 02:59:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 19 05:56:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 09b22869-e122-b881-0c3e-b9dc2cfbba0d (at 10.8.13.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ef28f2400, cur 1539953816 expire 1539953666 last 1539953589 Oct 19 06:57:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9fe3d0ed-2d31-1bb3-2b04-bb1764ccea1d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde1c400, cur 1539957462 expire 1539957312 last 1539957235 Oct 19 06:59:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 19 07:13:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:14:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1091e190-6432-e81f-2c05-135653896ceb (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ee807400, cur 1539958455 expire 1539958305 last 1539958228 Oct 19 07:17:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:17:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ca3b0eda-eb40-e429-80c5-53377fd9aba3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff4389800, cur 1539958676 expire 1539958526 last 1539958449 Oct 19 07:17:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 19 07:36:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e22cc77b-be45-b380-d541-19e69ef36bf2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ef8781800, cur 1539959766 expire 1539959616 last 1539959539 Oct 19 07:36:06 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 07:36:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:40:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:40:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0c72f3f-77fd-2b89-404a-a59d9ee23768 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880af70d4c00, cur 1539960046 expire 1539959896 last 1539959819 Oct 19 07:44:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52797aaa-5c18-1a5d-e20f-037802854a37 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8806dd91f800, cur 1539960267 expire 1539960117 last 1539960040 Oct 19 07:45:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:48:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:49:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 27ed05f1-fd6d-88c2-6d31-febe0f40d741 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ed2220400, cur 1539960564 expire 1539960414 last 1539960337 Oct 19 07:53:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79bdfc68-b560-d4e1-37ae-9ee7f7984fea (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881112f8a000, cur 1539960823 expire 1539960673 last 1539960596 Oct 19 07:55:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:58:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 07:59:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aba66633-1f64-6521-954f-7117dc9c8036 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f2a8000, cur 1539961165 expire 1539961015 last 1539960938 Oct 19 08:01:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Oct 19 08:02:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ab37ec9b-9460-2bfd-2051-460fa9a7c5a6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881698e95400, cur 1539961345 expire 1539961195 last 1539961118 Oct 19 08:07:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Oct 19 08:07:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 08:09:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b3beddd3-6be3-d998-6eca-bab727768d47 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800491b5400, cur 1539961764 expire 1539961614 last 1539961537 Oct 19 08:09:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 08:09:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 19 08:10:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e8eea999-d0c9-d590-8553-1629977887c6 (at 10.8.17.26@o2ib6) in 215 seconds. I think it's dead, and I am evicting it. exp ffff881250756c00, cur 1539961840 expire 1539961690 last 1539961625 Oct 19 08:10:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 08:12:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 08:12:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 08:15:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1953309b-0378-2b20-60d9-55204c69222b (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e3c20c00, cur 1539962137 expire 1539961987 last 1539961910 Oct 19 08:20:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d69a9acf-aa9a-22ae-29c7-b68c358bc5a5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c6932a400, cur 1539962402 expire 1539962252 last 1539962175 Oct 19 08:20:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 08:21:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 08:21:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 08:29:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5909c9da-ceda-c884-6a30-472c6f8dca55 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e59b17000, cur 1539962951 expire 1539962801 last 1539962724 Oct 19 08:29:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 08:31:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 19 08:31:55 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 08:42:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ad2c7f7e-85ed-168d-46eb-4d8ca7349504 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880046e2dc00, cur 1539963751 expire 1539963601 last 1539963524 Oct 19 08:42:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 08:44:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 08:44:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 08:59:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c4ce6bd8-0f3f-8a40-70c7-979c5fa90221 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ed256400, cur 1539964783 expire 1539964633 last 1539964556 Oct 19 08:59:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 09:02:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 09:02:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 19 09:13:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 09:13:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 09:14:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1faf7fac-214f-5571-5cc7-14827560871d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880582305800, cur 1539965657 expire 1539965507 last 1539965430 Oct 19 09:14:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 09:27:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c830ed5-4e70-ea5d-fee7-e5b8a616291a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880003708c00, cur 1539966428 expire 1539966278 last 1539966201 Oct 19 09:28:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 09:37:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93504282-778c-3021-84b1-2e23de85dd17 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815153a5000, cur 1539967031 expire 1539966881 last 1539966804 Oct 19 09:37:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 10:54:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b87853da-35f1-86bf-e5e8-2bbc6074d867 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff0b30c00, cur 1539971666 expire 1539971516 last 1539971439 Oct 19 11:21:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a816ffb-848a-a9f8-e400-c8e354652630 (at 10.8.14.7@o2ib6) Oct 19 11:21:26 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 11:33:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7373e52b-fb18-5bc9-cedb-53b9124a5b11 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacc58000, cur 1539974015 expire 1539973865 last 1539973788 Oct 19 12:01:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5336641-0c91-6c31-de53-aff0e58b665f (at 10.9.104.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b268d400, cur 1539975663 expire 1539975513 last 1539975436 Oct 19 12:10:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client caa27bf2-2493-d22d-1016-676fdbc540df (at 10.12.4.59@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103abb6800, cur 1539976240 expire 1539976090 last 1539976013 Oct 19 12:10:40 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 19 12:11:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c1c5ea59-40db-8ad9-fb5f-2c9fd5cf9542 (at 10.12.4.59@o2ib) Oct 19 12:20:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2afe4f35-8d56-c11a-28d1-d32bd8eb280a (at 10.12.4.36@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880071c08000, cur 1539976839 expire 1539976689 last 1539976612 Oct 19 12:22:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5228c5a7-eb77-96f5-1a27-5a3b661fd302 (at 10.9.104.2@o2ib4) Oct 19 12:22:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b5bece9f-1994-cc1e-a006-b95a2b0d4482 (at 10.9.104.1@o2ib4) Oct 19 12:22:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2ce6fa59-5e31-71ae-af70-fd0bf18fb38d (at 10.9.104.4@o2ib4) Oct 19 12:40:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Oct 19 12:42:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e18a9ae6-eec9-409c-29a7-72168b52d16f (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c5f6af800, cur 1539978140 expire 1539977990 last 1539977913 Oct 19 12:43:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65ca7133-a0fb-3b3d-4d0f-270ca4e58719 (at 10.8.15.3@o2ib6) in 181 seconds. I think it's dead, and I am evicting it. exp ffff880c6674b000, cur 1539978216 expire 1539978066 last 1539978035 Oct 19 13:02:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dea08e4f-a382-39df-b630-9ecb0fd52c25 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880225011000, cur 1539979370 expire 1539979220 last 1539979143 Oct 19 13:58:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4b449a98-1b92-bcc4-91c7-83fba376ae28 (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ab0862000, cur 1539982712 expire 1539982562 last 1539982485 Oct 19 14:16:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 19 14:17:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e0a19cc-212a-0db0-67c3-c9379868aba4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac96f800, cur 1539983842 expire 1539983692 last 1539983615 Oct 19 14:21:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 19 14:22:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cee8da61-f4b6-96c1-73fb-7def221f0f25 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ff3d98800, cur 1539984139 expire 1539983989 last 1539983912 Oct 19 14:22:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Oct 19 14:48:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 86b7dacc-4965-5783-8e68-656ca622ba28 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800537f5800, cur 1539985695 expire 1539985545 last 1539985468 Oct 19 14:48:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 19 15:01:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 83956553-e1a8-658b-e7d0-7b4ede503451 (at 10.9.107.19@o2ib4) Oct 19 15:24:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f03f8f14-89d5-6061-fc27-5de905920052 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a98400800, cur 1539987845 expire 1539987695 last 1539987618 Oct 19 15:48:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 19 15:59:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b58d668f-c0fd-757b-04cc-d7402b5932d1 (at 10.12.4.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817b3e10400, cur 1539989965 expire 1539989815 last 1539989738 Oct 19 16:00:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 919dfb2a-ca78-86a4-4022-cc063b1d1e79 (at 10.12.4.23@o2ib) Oct 19 16:01:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ac7dd2e6-957c-7138-459f-342637e2dd2b (at 10.12.4.58@o2ib) Oct 19 16:10:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da7e94a6-8ef3-9c80-b689-36014881dd8b (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b2f710800, cur 1539990648 expire 1539990498 last 1539990421 Oct 19 16:14:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9f378401-74eb-1805-a684-caacea7c7cd2 (at 10.9.107.37@o2ib4) Oct 19 16:50:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ab3aad7-1654-2b51-5652-7594fc5eddc0 (at 10.8.16.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db2f81800, cur 1539993041 expire 1539992891 last 1539992814 Oct 19 16:56:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d4bfb9b2-9492-885f-49d5-cf7fead04c0d (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881aa9bdcc00, cur 1539993410 expire 1539993260 last 1539993183 Oct 19 16:59:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 811615cb-712c-df21-7118-def9f2e565d1 (at 10.9.107.38@o2ib4) Oct 19 17:04:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 19 17:05:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f5ba14c4-40aa-32d0-4154-10cef5c6017d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880444f9c000, cur 1539993923 expire 1539993773 last 1539993696 Oct 19 18:56:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 19 18:56:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 181235dc-55c5-072d-eb2c-92f1f3c6fb3c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880004435000, cur 1540000597 expire 1540000447 last 1540000370 Oct 19 18:59:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 19 19:00:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5e403b82-6c34-50eb-0d06-fcc3eba5f67d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88183ae90400, cur 1540000845 expire 1540000695 last 1540000618 Oct 19 19:03:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e4455251-5a74-4972-65b6-18ca171dfc21 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88152da0c800, cur 1540001038 expire 1540000888 last 1540000811 Oct 19 19:11:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 19 19:12:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1f11f069-d734-0873-fef2-390b74539689 (at 10.8.10.9@o2ib6) reconnecting Oct 19 19:12:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b75866aa-418d-e1f3-5cbd-48d4dc354fb1 (at 10.8.10.9@o2ib6) Oct 19 19:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ead5f3c6-0ef7-64ca-fcef-9b87eefc66a2 (at 10.8.8.7@o2ib6) reconnecting Oct 19 19:13:23 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 19:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f6fdc9f9-7efb-b458-fa2b-ab061e01cbda (at 10.8.8.7@o2ib6) Oct 19 19:13:23 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 19 19:15:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 6c46d0b8-72fb-c42c-8552-169f384f9a40 (at 10.8.8.28@o2ib6) reconnecting Oct 19 19:15:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 56c644c5-922f-0c6b-a7ec-8f493c7e6a52 (at 10.8.6.25@o2ib6) Oct 19 19:15:25 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 19 19:47:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 15c45df1-17c8-57b3-7d43-907484f3c1b3 (at 10.8.27.3@o2ib6) reconnecting Oct 19 19:47:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 19:47:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 24f4fd6e-d5fd-d892-fb49-ac284f050442 (at 10.8.27.3@o2ib6) Oct 19 19:47:15 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 19 19:47:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 9627849e-c83c-1587-29ba-dd885626608b (at 10.8.17.13@o2ib6) reconnecting Oct 19 19:47:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 19:47:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0bcbc27e-a2ea-ec0f-4344-c2297f0c44ae (at 10.8.17.13@o2ib6) Oct 19 19:47:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 19:54:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 005b3261-93c0-7f54-a380-e78e4ec29cb8 (at 10.8.27.10@o2ib6) reconnecting Oct 19 19:54:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14b8d305-ce8c-ddcf-a413-3e35caffc922 (at 10.8.27.10@o2ib6) Oct 19 20:21:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 6bca5236-f4ce-38b7-69f7-3368a86bcbf6 (at 10.8.1.10@o2ib6) reconnecting Oct 19 20:21:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8d509be9-9142-39b7-5021-3ec3c3e61002 (at 10.8.1.10@o2ib6) Oct 19 20:21:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ac23ddbe-bf02-bdd5-6d87-8047ca89715f (at 10.8.8.24@o2ib6) reconnecting Oct 19 20:21:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3962ccdf-e186-6c74-a165-d8fbbf8c5c4d (at 10.8.8.24@o2ib6) Oct 19 20:22:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ac23ddbe-bf02-bdd5-6d87-8047ca89715f (at 10.8.8.24@o2ib6) reconnecting Oct 19 20:22:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3962ccdf-e186-6c74-a165-d8fbbf8c5c4d (at 10.8.8.24@o2ib6) Oct 19 20:25:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 642fdfa1-0004-31c0-be9f-c7752b81b968 (at 10.8.8.2@o2ib6) reconnecting Oct 19 20:25:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f72c4ffd-cb8b-85d4-fc78-f2d5a62e22c4 (at 10.8.8.2@o2ib6) Oct 19 20:46:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 489f6b62-b28c-e7c7-405d-7fa6c3ca5a17 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881a1ee33c00, cur 1540007214 expire 1540007064 last 1540006987 Oct 19 21:07:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 236b5f06-7386-0d2d-1e58-63072b559415 (at 10.8.17.6@o2ib6) reconnecting Oct 19 21:07:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2455eb3f-12df-a8cc-df9f-a0917ba1883c (at 10.8.1.18@o2ib6) Oct 19 21:07:30 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 21:17:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 005b3261-93c0-7f54-a380-e78e4ec29cb8 (at 10.8.27.10@o2ib6) reconnecting Oct 19 21:17:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 14b8d305-ce8c-ddcf-a413-3e35caffc922 (at 10.8.27.10@o2ib6) Oct 19 21:17:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 21:30:08 oak-md1-s2 kernel: Lustre: 4255:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540009801/real 1540009801] req@ffff880121733600 x1610633735021184/t0(0) o104->oak-MDT0000@10.8.2.2@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540009808 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 19 21:30:08 oak-md1-s2 kernel: Lustre: 4255:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 19 21:30:15 oak-md1-s2 kernel: Lustre: 4255:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540009808/real 1540009808] req@ffff880121733600 x1610633735021184/t0(0) o104->oak-MDT0000@10.8.2.2@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540009815 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 19 21:30:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7275b786-d068-a9b4-85d5-d4aed73a25e6 (at 10.8.2.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9c400, cur 1540009819 expire 1540009669 last 1540009592 Oct 19 21:49:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 41b6a882-77d7-3b6d-714e-ad9a1459fe31 (at 10.8.6.21@o2ib6) reconnecting Oct 19 21:49:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d296a15e-8154-8527-be08-b302d3e42b22 (at 10.8.6.21@o2ib6) Oct 19 22:00:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8eda02b9-f2c3-9c35-b13e-5ed8b63783ea (at 10.8.6.9@o2ib6) reconnecting Oct 19 22:00:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1db58f68-7c4f-2629-6fce-82554b89fc05 (at 10.8.6.9@o2ib6) Oct 19 22:03:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e0fa4477-4729-93b4-faa3-3098d77afb8d (at 10.8.4.32@o2ib6) reconnecting Oct 19 22:03:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c5deefe8-6878-a928-038f-63911141e46c (at 10.8.4.32@o2ib6) Oct 19 22:05:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e0fa4477-4729-93b4-faa3-3098d77afb8d (at 10.8.4.32@o2ib6) reconnecting Oct 19 22:05:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c5deefe8-6878-a928-038f-63911141e46c (at 10.8.4.32@o2ib6) Oct 19 22:06:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client b766a494-a85e-55a0-758f-55e40391d3ef (at 10.8.3.2@o2ib6) reconnecting Oct 19 22:06:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 585d8ff8-4575-4ba4-5af4-02aa57043521 (at 10.8.3.2@o2ib6) Oct 19 22:36:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0e45fbed-5d84-4f37-92da-b7be85a10ca5 (at 10.8.6.10@o2ib6) reconnecting Oct 19 22:36:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f52d7fa9-a491-9d23-3523-b1afd1fdf6f0 (at 10.8.6.10@o2ib6) Oct 19 22:37:01 oak-md1-s2 kernel: Lustre: 10560:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540013814/real 1540013814] req@ffff881460733900 x1610633753688192/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540013821 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 19 22:37:08 oak-md1-s2 kernel: Lustre: 10560:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540013821/real 1540013821] req@ffff881460733900 x1610633753688192/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540013828 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 19 22:37:15 oak-md1-s2 kernel: Lustre: 10560:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540013828/real 1540013828] req@ffff881460733900 x1610633753688192/t0(0) o104->oak-MDT0000@10.8.3.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540013835 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 19 22:37:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 95fae0f2-3f98-64b0-a1d7-22c8b5a2edda (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88181977fc00, cur 1540013841 expire 1540013691 last 1540013614 Oct 19 22:38:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99c5bbf8-f493-a0c9-2e8d-4f60faedac1a (at 10.8.3.27@o2ib6) Oct 19 22:39:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 4983501e-cfde-0077-1d3b-d0c2af2f675c (at 10.8.4.20@o2ib6) reconnecting Oct 19 22:39:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f2874060-6bfb-c6c5-c8a8-60d8ca27c3be (at 10.8.6.7@o2ib6) Oct 19 22:39:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 22:54:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e0fa4477-4729-93b4-faa3-3098d77afb8d (at 10.8.4.32@o2ib6) reconnecting Oct 19 22:54:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c5deefe8-6878-a928-038f-63911141e46c (at 10.8.4.32@o2ib6) Oct 19 22:54:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 22:55:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 4de58143-1b44-8562-d1ed-c03503d358fa (at 10.8.4.19@o2ib6) reconnecting Oct 19 22:55:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b067ccdd-27cb-ae5c-27d9-df62cb606375 (at 10.8.4.19@o2ib6) Oct 19 22:56:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client b766a494-a85e-55a0-758f-55e40391d3ef (at 10.8.3.2@o2ib6) reconnecting Oct 19 22:56:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 585d8ff8-4575-4ba4-5af4-02aa57043521 (at 10.8.3.2@o2ib6) Oct 19 22:56:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 64b91601-7f74-fc13-c790-2fd05e794761 (at 10.8.4.33@o2ib6) reconnecting Oct 19 22:56:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2eb5c032-e495-d959-c1b3-624fdefab331 (at 10.8.4.33@o2ib6) Oct 19 23:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8eda02b9-f2c3-9c35-b13e-5ed8b63783ea (at 10.8.6.9@o2ib6) reconnecting Oct 19 23:02:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1db58f68-7c4f-2629-6fce-82554b89fc05 (at 10.8.6.9@o2ib6) Oct 19 23:03:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 4de58143-1b44-8562-d1ed-c03503d358fa (at 10.8.4.19@o2ib6) reconnecting Oct 19 23:03:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b067ccdd-27cb-ae5c-27d9-df62cb606375 (at 10.8.4.19@o2ib6) Oct 19 23:04:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client d966ccc9-c5fd-d4ad-a64e-8e6cebf7c5b0 (at 10.8.6.13@o2ib6) reconnecting Oct 19 23:04:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4db94d05-0dab-919f-3011-c5b8ea415e50 (at 10.8.6.13@o2ib6) Oct 19 23:09:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 32d396a9-1c2a-13e2-aa76-49378b797751 (at 10.8.4.28@o2ib6) reconnecting Oct 19 23:09:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8624f01e-1d3e-4405-7476-6476c124e6e7 (at 10.8.4.28@o2ib6) Oct 19 23:12:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 128b3170-3a01-9359-6a43-e499aeab0560 (at 10.8.4.18@o2ib6) reconnecting Oct 19 23:12:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c5deefe8-6878-a928-038f-63911141e46c (at 10.8.4.32@o2ib6) Oct 19 23:12:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 19 23:18:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 128b3170-3a01-9359-6a43-e499aeab0560 (at 10.8.4.18@o2ib6) reconnecting Oct 19 23:18:48 oak-md1-s2 kernel: Lustre: Skipped 54 previous similar messages Oct 19 23:18:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3796746c-13cb-1dff-031b-f34216a2f802 (at 10.8.4.18@o2ib6) Oct 19 23:18:48 oak-md1-s2 kernel: Lustre: Skipped 55 previous similar messages Oct 19 23:24:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f2b4dd3c-2897-f12d-1195-18fd03df4a02 (at 10.8.6.2@o2ib6) reconnecting Oct 19 23:24:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1db58f68-7c4f-2629-6fce-82554b89fc05 (at 10.8.6.9@o2ib6) Oct 19 23:24:01 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 00:30:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2894e7b-88f3-f12b-9972-fbab31860e81 (at 10.9.101.47@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bbf1dfc00, cur 1540020649 expire 1540020499 last 1540020422 Oct 20 00:32:28 oak-md1-s2 kernel: LustreError: 5044:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.47@o2ib4 arrived at 1540020748 with bad export cookie 7782435640500901040 Oct 20 00:32:28 oak-md1-s2 kernel: LustreError: 5044:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) Skipped 3080 previous similar messages Oct 20 00:32:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb00c1ca-a02c-8822-a1bb-ad0f0429fe24 (at 10.9.101.47@o2ib4) Oct 20 00:32:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 00:36:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2894e7b-88f3-f12b-9972-fbab31860e81 (at 10.9.101.47@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818893ec400, cur 1540020975 expire 1540020825 last 1540020748 Oct 20 00:37:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7604b37-cfd5-836d-ce2f-93f427020c9a (at 10.9.101.55@o2ib4) in 155 seconds. I think it's dead, and I am evicting it. exp ffff881fe7400800, cur 1540021051 expire 1540020901 last 1540020896 Oct 20 01:30:33 oak-md1-s2 kernel: LustreError: 260427:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.9.101.55@o2ib4 arrived at 1540024233 with bad export cookie 7782435640501715392 Oct 20 01:30:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fd226631-ec42-ea48-1eb1-b60097c20b51 (at 10.9.101.55@o2ib4) Oct 20 01:34:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b7604b37-cfd5-836d-ce2f-93f427020c9a (at 10.9.101.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881856ff5800, cur 1540024460 expire 1540024310 last 1540024233 Oct 20 01:47:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client de07c8e1-4bda-8eb0-5553-cb64e1b7a646 (at 10.8.6.15@o2ib6) reconnecting Oct 20 01:47:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5fb42f44-6d20-1500-290f-13e593b5bb24 (at 10.8.6.15@o2ib6) Oct 20 02:11:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d908bf98-8428-6c9e-781e-8fdd365c86be (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f8a3bf000, cur 1540026714 expire 1540026564 last 1540026487 Oct 20 02:14:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb00c1ca-a02c-8822-a1bb-ad0f0429fe24 (at 10.9.101.47@o2ib4) Oct 20 02:15:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 02:20:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c881c31-fb93-e5e4-57b4-3961e7c581ee (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816ca14c800, cur 1540027213 expire 1540027063 last 1540026986 Oct 20 02:20:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 02:26:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc8f7883-24b2-e15a-85ac-dbf1c9992320 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88073018b400, cur 1540027566 expire 1540027416 last 1540027339 Oct 20 02:27:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 02:31:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ffc89681-a789-09b4-6946-872b6716ae7e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881966f50400, cur 1540027911 expire 1540027761 last 1540027684 Oct 20 02:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 02:44:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2dea985c-1a91-5ae7-18af-729128f45985 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b643d000, cur 1540028664 expire 1540028514 last 1540028437 Oct 20 02:46:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 02:49:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 920bd62c-5cc4-bb78-1149-bbe65f49f410 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a2800, cur 1540028994 expire 1540028844 last 1540028767 Oct 20 02:51:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 03:01:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b352ff74-df16-dbf3-f373-e44fe5596901 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200befb000, cur 1540029665 expire 1540029515 last 1540029438 Oct 20 03:24:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6ed46e60-6537-9ebd-761b-e36650f93d6d (at 10.12.4.38@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880810d0e800, cur 1540031067 expire 1540030917 last 1540030840 Oct 20 03:25:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 20 03:25:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 20 03:39:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 548a76e1-038d-9cda-4b2e-54322a96fce1 (at 10.12.4.73@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e30e39800, cur 1540031966 expire 1540031816 last 1540031739 Oct 20 05:05:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 20 05:05:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 20 05:05:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 20 05:05:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 20 05:10:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:14:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:15:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 27390953-7780-7da1-a89f-c4446d7df42e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88113a715400, cur 1540037703 expire 1540037553 last 1540037476 Oct 20 05:18:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:18:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client eaf0f163-8965-42d1-2a4d-e3ff31fd39a6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ad9e8000, cur 1540037936 expire 1540037786 last 1540037709 Oct 20 05:22:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:22:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1845e1e9-f5e3-f482-dd6d-5140c3fabf04 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881aa9bdcc00, cur 1540038174 expire 1540038024 last 1540037947 Oct 20 05:26:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b22a595-c744-6aa6-f31e-9bfc4edc3d1e (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff0b37c00, cur 1540038411 expire 1540038261 last 1540038184 Oct 20 05:29:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:33:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:33:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e999ab73-31c4-3340-8c1f-58a598e33a05 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031dc72000, cur 1540038816 expire 1540038666 last 1540038589 Oct 20 05:36:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:37:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 547495c8-f625-8346-35c4-507b266a257f (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8820107e8400, cur 1540039052 expire 1540038902 last 1540038825 Oct 20 05:40:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:41:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9056e2e5-b429-d4ae-ad08-d5bc4ff16bf6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d094b8800, cur 1540039280 expire 1540039130 last 1540039053 Oct 20 05:45:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:45:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8401c6ff-f9c4-9282-a42f-d3c62b9c5129 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88002715e400, cur 1540039504 expire 1540039354 last 1540039277 Oct 20 05:49:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dac488c7-c90d-6123-35a4-f73e6b14a76d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b7978ec00, cur 1540039763 expire 1540039613 last 1540039536 Oct 20 05:53:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 86b99602-c73a-059d-2392-67f26aef58b7 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880814644000, cur 1540039998 expire 1540039848 last 1540039771 Oct 20 05:59:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 05:59:19 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 05:59:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3b388a04-65bd-1bbd-4466-064c23823cc9 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88061e88b400, cur 1540040397 expire 1540040247 last 1540040170 Oct 20 06:17:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf646e22-6908-6846-adf3-af7bb4866676 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e3c22000, cur 1540041428 expire 1540041278 last 1540041201 Oct 20 06:17:08 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 06:17:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 06:17:43 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 07:10:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04ad4a1a-8600-713d-f117-9941ab5c50bf (at 10.12.4.43@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000370e800, cur 1540044642 expire 1540044492 last 1540044415 Oct 20 07:10:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 07:13:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7e03d195-0965-9222-f1e5-b7590cae3067 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8813616e3c00, cur 1540044802 expire 1540044652 last 1540044575 Oct 20 07:14:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 07:14:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 07:17:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 07:18:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ea2fd842-85b8-75ef-59a9-775e98308617 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e72c56400, cur 1540045122 expire 1540044972 last 1540044895 Oct 20 07:22:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 07:27:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7feba8f8-598e-5408-2b91-63a2e40de517 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881090521000, cur 1540045623 expire 1540045473 last 1540045396 Oct 20 07:27:03 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 07:31:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 07:31:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 07:43:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 07:43:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 07:44:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52f7de9c-de28-d63f-4dc1-984191b612c4 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817a83aa400, cur 1540046646 expire 1540046496 last 1540046419 Oct 20 07:44:06 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 07:58:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59587b39-f82b-f69f-6549-384e6ab5ce41 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe1343800, cur 1540047511 expire 1540047361 last 1540047284 Oct 20 07:58:31 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 08:04:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 08:04:50 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 08:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 08:09:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c282fad8-98f7-4ca3-cc80-bda5020a892d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880067541400, cur 1540048192 expire 1540048042 last 1540047965 Oct 20 08:18:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 08:24:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b236445-32b5-4d26-b5b5-9ef8d784003a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d0947bc00, cur 1540049069 expire 1540048919 last 1540048842 Oct 20 08:24:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 08:30:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 08:40:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 08:40:47 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 08:41:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 558786e1-8b5f-aae5-0e34-57ce86d88241 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880018816400, cur 1540050098 expire 1540049948 last 1540049871 Oct 20 08:41:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 08:52:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 08:52:48 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 08:57:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be99c82b-41b7-ba96-0a74-ab80ab4f5879 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b3a38400, cur 1540051029 expire 1540050879 last 1540050802 Oct 20 08:57:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 09:32:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e5719adb-05ee-a46f-8156-65f30e27a493 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b67284800, cur 1540053169 expire 1540053019 last 1540052942 Oct 20 09:35:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 09:35:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 09:38:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 09:39:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ae9ba52-ad58-b177-1a48-64d3cbc596e2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e75231800, cur 1540053565 expire 1540053415 last 1540053338 Oct 20 09:43:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 924a017b-8897-7c15-8dfc-1172f31688b2 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe937d000, cur 1540053801 expire 1540053651 last 1540053574 Oct 20 09:43:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 09:49:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fd3cabe1-bb62-552e-ed93-77d2b2f6e043 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880810d0a400, cur 1540054154 expire 1540054004 last 1540053927 Oct 20 09:58:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 10:03:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1eca44c3-290b-896a-6d87-905dfd931fb6 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814f774bc00, cur 1540055036 expire 1540054886 last 1540054809 Oct 20 10:52:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 10:52:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 10:53:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e128da4a-6c24-599d-fddd-c854eb52f396 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88113a716c00, cur 1540058007 expire 1540057857 last 1540057780 Oct 20 10:56:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 10:57:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 74a499fa-3689-e886-5243-f5661e8e903a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003a19f800, cur 1540058232 expire 1540058082 last 1540058005 Oct 20 11:00:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 11:01:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e86aa976-9b3e-d76a-63f4-7e3323155595 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88018dd70c00, cur 1540058478 expire 1540058328 last 1540058251 Oct 20 11:18:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3984dfc7-78a6-45b4-f9f8-206d482f0982 (at 10.12.4.70@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880343110c00, cur 1540059490 expire 1540059340 last 1540059263 Oct 20 11:18:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 11:35:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 994f18d1-7839-d8dd-32e8-f0ee3ee6df22 (at 10.8.8.13@o2ib6) reconnecting Oct 20 11:35:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f1082753-e805-bcf4-6a35-159c3f296b00 (at 10.8.8.13@o2ib6) Oct 20 11:36:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 994f18d1-7839-d8dd-32e8-f0ee3ee6df22 (at 10.8.8.13@o2ib6) reconnecting Oct 20 11:36:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f1082753-e805-bcf4-6a35-159c3f296b00 (at 10.8.8.13@o2ib6) Oct 20 12:51:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aee68f6e-0c75-311d-92a9-00b3de5c2a33 (at 10.12.4.45@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88004b634000, cur 1540065112 expire 1540064962 last 1540064885 Oct 20 12:55:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48cf3906-7c03-1c84-c798-92e4ed94c74b (at 10.8.7.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881893b8ec00, cur 1540065322 expire 1540065172 last 1540065095 Oct 20 12:56:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2e2a365f-3b8d-6fd1-d503-3c4e36e6c5fc (at 10.8.7.18@o2ib6) Oct 20 13:57:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client be0dbdd5-12be-af7d-5091-fbb3b4d6ed20 (at 10.8.8.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb213f400, cur 1540069039 expire 1540068889 last 1540068812 Oct 20 13:58:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b7194205-b2c9-bce5-f35b-d3d4bc5723ab (at 10.8.8.10@o2ib6) Oct 20 15:17:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04f156a8-2297-69fe-3124-5343d475064a (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881551b3d800, cur 1540073868 expire 1540073718 last 1540073641 Oct 20 15:18:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 20 15:25:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 306894b5-e539-ad88-b61f-d164d7ae5a9f (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800196b5400, cur 1540074311 expire 1540074161 last 1540074084 Oct 20 15:28:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 20 15:37:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6644a5e9-6fc3-490b-4105-0edeca7fcc53 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88108fa01000, cur 1540075052 expire 1540074902 last 1540074825 Oct 20 16:26:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fc0b6455-60a3-8a66-4d2f-3bd526333db4 (at 10.12.4.58@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e47400, cur 1540077966 expire 1540077816 last 1540077739 Oct 20 16:27:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ab6b185d-25f8-c381-7540-a1076f8ace0c (at 10.8.7.25@o2ib6) Oct 20 16:28:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b76d645f-ad85-4398-98c5-055f3721fde3 (at 10.8.7.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a304b9800, cur 1540078122 expire 1540077972 last 1540077895 Oct 20 22:49:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 22:54:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 22:54:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1e84199c-139a-6fb4-e503-b18e7686c79d (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b643e800, cur 1540101256 expire 1540101106 last 1540101029 Oct 20 22:58:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 22:59:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 37027eff-4960-63cd-f668-e4b78be30ecc (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9e800, cur 1540101547 expire 1540101397 last 1540101320 Oct 20 23:00:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4ea6023-2989-a704-ff50-25d1efb64007 (at 10.8.14.1@o2ib6) in 160 seconds. I think it's dead, and I am evicting it. exp ffff881812766c00, cur 1540101623 expire 1540101473 last 1540101463 Oct 20 23:01:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 20 23:01:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 20 23:01:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a15c6640-d1af-7a09-1f27-713b32725465 (at 10.8.17.26@o2ib6) in 162 seconds. I think it's dead, and I am evicting it. exp ffff88186c672400, cur 1540101699 expire 1540101549 last 1540101537 Oct 20 23:02:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 20 23:02:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 20 23:02:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 23:06:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 23:07:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b43fe843-40e2-f1ce-e333-d0cba73b91e3 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880727407c00, cur 1540102044 expire 1540101894 last 1540101817 Oct 20 23:10:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 23:11:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d5a952c6-a633-4b25-776b-ca2606bcc4a5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096a0f7400, cur 1540102302 expire 1540102152 last 1540102075 Oct 20 23:15:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 23:15:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ad6c7c04-9882-1281-fbd0-b96cd537a6d5 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf938800, cur 1540102551 expire 1540102401 last 1540102324 Oct 20 23:30:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 730923af-b646-7905-4eac-f9cc45cd1844 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880015a75800, cur 1540103400 expire 1540103250 last 1540103173 Oct 20 23:32:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 23:42:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c59730c5-4c53-d930-3282-fc3a339c2d69 (at 10.8.27.25@o2ib6) reconnecting Oct 20 23:42:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 790c6f8f-26ec-41ff-6eef-2fa119576e77 (at 10.8.26.4@o2ib6) Oct 20 23:42:45 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 23:43:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 426aaed5-1677-624c-ab0b-ff023557cca9 (at 10.8.3.24@o2ib6) reconnecting Oct 20 23:43:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client de07c8e1-4bda-8eb0-5553-cb64e1b7a646 (at 10.8.6.15@o2ib6) reconnecting Oct 20 23:43:19 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 20 23:43:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5fb42f44-6d20-1500-290f-13e593b5bb24 (at 10.8.6.15@o2ib6) Oct 20 23:43:19 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 20 23:43:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e67fa40f-b5a4-3c97-8a60-c1eca96ae03f (at 10.8.4.17@o2ib6) reconnecting Oct 20 23:43:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 23:44:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8891d6fd-0245-fb65-bc14-95cc277c7a1a (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881855b71000, cur 1540104240 expire 1540104090 last 1540104013 Oct 20 23:45:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4f13f3a-4d99-fe71-f01b-bacd0f1e1bd0 (at 10.8.17.26@o2ib6) Oct 20 23:45:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 23:45:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e3d3b220-6be1-d59b-091e-127e9ee8cbf9 (at 10.8.3.14@o2ib6) reconnecting Oct 20 23:46:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 94fa2938-5a17-2f3e-8f59-2b04fc1032e8 (at 10.8.3.15@o2ib6) reconnecting Oct 20 23:47:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 61d93ab5-3ad0-d414-c5b4-a6ed3bbcf74a (at 10.8.3.5@o2ib6) reconnecting Oct 20 23:47:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2012ec85-a6fc-f7ce-e763-43d3a322b64e (at 10.8.3.14@o2ib6) Oct 20 23:47:30 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 20 23:47:30 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 23:48:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2c5f7722-4fad-d38d-f918-0a4dcb48f218 (at 10.8.2.7@o2ib6) reconnecting Oct 20 23:48:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 20 23:49:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 5973c953-1913-9ec2-4f29-2d1a6dbc44db (at 10.8.4.1@o2ib6) reconnecting Oct 20 23:49:40 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 20 23:53:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client b5646a3a-c63c-96bd-3ad0-a48107497162 (at 10.8.3.30@o2ib6) reconnecting Oct 20 23:53:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 20 23:53:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 64709e3d-aec3-8528-aefd-ad77d000a881 (at 10.8.3.30@o2ib6) Oct 20 23:53:14 oak-md1-s2 kernel: Lustre: Skipped 10 previous similar messages Oct 20 23:57:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f53687fe-917d-4f1d-f88d-4e2752c8aa5c (at 10.8.6.23@o2ib6) reconnecting Oct 20 23:57:54 oak-md1-s2 kernel: Lustre: Skipped 79 previous similar messages Oct 21 00:01:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8d877739-3c8c-7272-f8cd-ac7286dc8802 (at 10.8.17.10@o2ib6) Oct 21 00:01:47 oak-md1-s2 kernel: Lustre: Skipped 131 previous similar messages Oct 21 00:07:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 61d93ab5-3ad0-d414-c5b4-a6ed3bbcf74a (at 10.8.3.5@o2ib6) reconnecting Oct 21 00:07:09 oak-md1-s2 kernel: Lustre: Skipped 191 previous similar messages Oct 21 00:11:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9be6edfa-ebde-1d5d-f312-2249d8fda40b (at 10.8.2.11@o2ib6) Oct 21 00:11:48 oak-md1-s2 kernel: Lustre: Skipped 234 previous similar messages Oct 21 00:11:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 97fb035b-f73d-2e4d-7b48-09b033515879 (at 10.8.17.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e45800, cur 1540105911 expire 1540105761 last 1540105684 Oct 21 00:19:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1e67908e-bee8-83ba-9aee-70ca8d4b8287 (at 10.8.6.8@o2ib6) reconnecting Oct 21 00:19:52 oak-md1-s2 kernel: Lustre: Skipped 198 previous similar messages Oct 21 00:20:57 oak-md1-s2 kernel: LustreError: 10535:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.4.1@o2ib6) returned error from blocking AST (req@ffff881d49689800 x1610634565361728 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8803b2037800/0x6c00c430dff87a06 lrc: 4/0,0 mode: PR/PR res: [0x2000182ac:0x1c26b:0x0].0x0 bits 0x20 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.4.1@o2ib6 remote: 0x52519f60cb315653 expref: 53189 pid: 10552 timeout: 8391550591 lvb_type: 0 Oct 21 00:20:57 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.4.1@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 00:21:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2012ec85-a6fc-f7ce-e763-43d3a322b64e (at 10.8.3.14@o2ib6) Oct 21 00:21:49 oak-md1-s2 kernel: Lustre: Skipped 172 previous similar messages Oct 21 00:22:14 oak-md1-s2 kernel: LustreError: 10638:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.3.12@o2ib6) returned error from blocking AST (req@ffff880becb94500 x1610634566878304 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881e00575200/0x6c00c430e74dfbe6 lrc: 4/0,0 mode: PR/PR res: [0x2000182ca:0x124f:0x0].0x0 bits 0x20 rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.3.12@o2ib6 remote: 0x747a52cbbecfaa44 expref: 40971 pid: 10633 timeout: 8391627494 lvb_type: 0 Oct 21 00:22:14 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.3.12@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 00:23:15 oak-md1-s2 kernel: LustreError: 10538:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.3.14@o2ib6) returned error from blocking AST (req@ffff881e21607b00 x1610634568181392 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff88004f7c9600/0x6c00c430ed654475 lrc: 4/0,0 mode: CR/CR res: [0x2000182a9:0x1ecec:0x0].0x0 bits 0x9 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.8.3.14@o2ib6 remote: 0x63c8a16e71df3f56 expref: 53365 pid: 10590 timeout: 8390888133 lvb_type: 0 Oct 21 00:23:15 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.3.14@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 00:23:16 oak-md1-s2 kernel: LustreError: 11708:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88203ab2c050 x1613610295879904/t0(0) o37->5973c953-1913-9ec2-4f29-2d1a6dbc44db@10.8.4.1@o2ib6:222/0 lens 568/440 e 0 to 0 dl 1540106602 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 00:23:30 oak-md1-s2 kernel: LustreError: 7147:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880331333f00 x1613512930759552/t0(0) o37->eeb4b74f-549f-d327-e480-81ac5f6e9780@10.8.18.4@o2ib6:237/0 lens 568/440 e 0 to 0 dl 1540106617 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 00:23:50 oak-md1-s2 kernel: LustreError: 12983:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881daf204850 x1613525219959664/t0(0) o37->a056288f-afe6-ee33-35e6-2b072f4d7956@10.8.7.20@o2ib6:257/0 lens 568/440 e 0 to 0 dl 1540106637 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 00:29:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f0b5d0b6-07ca-2779-b47b-2586de460798 (at 10.8.3.32@o2ib6) reconnecting Oct 21 00:29:59 oak-md1-s2 kernel: Lustre: Skipped 341 previous similar messages Oct 21 00:30:42 oak-md1-s2 kernel: LustreError: 10659:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.3.7@o2ib6) returned error from blocking AST (req@ffff8804bac17500 x1610634577910944 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8810b23bd800/0x6c00c42f7a784c0c lrc: 4/0,0 mode: PR/PR res: [0x200017b1d:0x17fe0:0x0].0x0 bits 0x1b rrc: 5 type: IBT flags: 0x60200400000020 nid: 10.8.3.7@o2ib6 remote: 0x9ed97c19a2a23118 expref: 52054 pid: 10619 timeout: 8391335266 lvb_type: 0 Oct 21 00:30:42 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.3.7@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 00:31:34 oak-md1-s2 kernel: LustreError: 10531:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.3.22@o2ib6) returned error from blocking AST (req@ffff8808a8c53600 x1610634579034576 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881bc4629c00/0x6c00c4311df01eca lrc: 4/0,0 mode: CR/CR res: [0x200018242:0x12ded:0x0].0x0 bits 0x9 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.3.22@o2ib6 remote: 0xb0e02f832de20803 expref: 23596 pid: 10575 timeout: 8391387785 lvb_type: 0 Oct 21 00:31:34 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.3.22@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 00:31:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d049b869-d728-0a9c-64a8-4c2cb1a8428f (at 10.8.3.7@o2ib6) Oct 21 00:31:49 oak-md1-s2 kernel: Lustre: Skipped 353 previous similar messages Oct 21 00:39:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8b151fd9-d625-1ae6-a556-e0591d9049fd (at 10.8.1.34@o2ib6) reconnecting Oct 21 00:39:59 oak-md1-s2 kernel: Lustre: Skipped 457 previous similar messages Oct 21 00:41:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7b2bbe73-105d-158e-e95e-f81a329398cf (at 10.8.3.31@o2ib6) Oct 21 00:41:54 oak-md1-s2 kernel: Lustre: Skipped 446 previous similar messages Oct 21 00:44:17 oak-md1-s2 kernel: LustreError: 7166:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880b1323e850 x1613929286271616/t0(0) o37->f0b5d0b6-07ca-2779-b47b-2586de460798@10.8.3.32@o2ib6:9/0 lens 568/440 e 0 to 0 dl 1540107899 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 00:50:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 426aaed5-1677-624c-ab0b-ff023557cca9 (at 10.8.3.24@o2ib6) reconnecting Oct 21 00:50:50 oak-md1-s2 kernel: Lustre: Skipped 373 previous similar messages Oct 21 00:51:46 oak-md1-s2 kernel: LustreError: 181528:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8815f3d9b050 x1614594749789904/t0(0) o37->e3d3b220-6be1-d59b-091e-127e9ee8cbf9@10.8.3.14@o2ib6:459/0 lens 568/440 e 0 to 0 dl 1540108349 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 00:51:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to efd35555-5fc0-7307-e511-6d9f56f7bb12 (at 10.8.3.5@o2ib6) Oct 21 00:51:59 oak-md1-s2 kernel: Lustre: Skipped 340 previous similar messages Oct 21 00:54:54 oak-md1-s2 kernel: LustreError: 10532:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.3.7@o2ib6) returned error from blocking AST (req@ffff88143bea9e00 x1610634604098128 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880c49e99600/0x6c00c431aa6c1cd6 lrc: 4/0,0 mode: CR/CR res: [0x2000182b6:0x18ae2:0x0].0x0 bits 0x9 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.3.7@o2ib6 remote: 0x9ed97c19a3e38918 expref: 52832 pid: 4295 timeout: 8393587320 lvb_type: 0 Oct 21 00:54:54 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.3.7@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 01:01:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 3c584b6c-619a-5284-c2d7-34b6f585c3ba (at 10.8.4.31@o2ib6) reconnecting Oct 21 01:01:57 oak-md1-s2 kernel: Lustre: Skipped 474 previous similar messages Oct 21 01:02:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd150062-e87a-eb7f-5d60-b6eab8819537 (at 10.8.4.5@o2ib6) Oct 21 01:02:22 oak-md1-s2 kernel: Lustre: Skipped 439 previous similar messages Oct 21 01:06:40 oak-md1-s2 kernel: LustreError: 10638:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.3.15@o2ib6) returned error from blocking AST (req@ffff8803b2539e00 x1610634617229376 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880df3fbfa00/0x6c00c431f17c32aa lrc: 4/0,0 mode: CR/CR res: [0x2000182c0:0x1779d:0x0].0x0 bits 0x9 rrc: 4 type: IBT flags: 0x60200400000020 nid: 10.8.3.15@o2ib6 remote: 0x26903f360b1b4fae expref: 55723 pid: 3871 timeout: 8393493346 lvb_type: 0 Oct 21 01:06:40 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.3.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 01:10:58 oak-md1-s2 kernel: LustreError: 3875:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff881d6793d050 x1613780603943072/t0(0) o37->128b3170-3a01-9359-6a43-e499aeab0560@10.8.4.18@o2ib6:64/0 lens 568/440 e 0 to 0 dl 1540109464 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 01:11:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5973c953-1913-9ec2-4f29-2d1a6dbc44db (at 10.8.4.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803fde18c00, cur 1540109467 expire 1540109317 last 1540109240 Oct 21 01:11:07 oak-md1-s2 kernel: LustreError: 10558:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880ce5ee8900 x1610634621131840/t0(0) o104->oak-MDT0000@10.8.4.1@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 21 01:11:07 oak-md1-s2 kernel: LustreError: 10558:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 22 previous similar messages Oct 21 01:11:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client d11fc576-8f1e-9c3e-4e16-749f9e3ae687 (at 10.8.4.10@o2ib6) reconnecting Oct 21 01:11:59 oak-md1-s2 kernel: Lustre: Skipped 314 previous similar messages Oct 21 01:12:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6cdfa4a1-0eb5-fc32-84dc-5732f06d5d85 (at 10.8.4.7@o2ib6) Oct 21 01:12:27 oak-md1-s2 kernel: Lustre: Skipped 321 previous similar messages Oct 21 01:22:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 41b6a882-77d7-3b6d-714e-ad9a1459fe31 (at 10.8.6.21@o2ib6) reconnecting Oct 21 01:22:09 oak-md1-s2 kernel: Lustre: Skipped 326 previous similar messages Oct 21 01:22:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d16a09c-6a22-6d4f-e06b-9b733db70a9d (at 10.8.7.35@o2ib6) Oct 21 01:22:28 oak-md1-s2 kernel: Lustre: Skipped 353 previous similar messages Oct 21 01:30:59 oak-md1-s2 kernel: LustreError: 7136:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880dd9a42100 x1613284227818272/t0(0) o37->32a9c517-00a7-8529-8354-731d186580b7@10.8.2.11@o2ib6:547/0 lens 568/440 e 0 to 0 dl 1540110702 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 01:32:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 92d9fc34-4c3d-e502-1d9b-f70cc1909dae (at 10.8.3.7@o2ib6) reconnecting Oct 21 01:32:16 oak-md1-s2 kernel: Lustre: Skipped 252 previous similar messages Oct 21 01:32:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 51435d4a-c2fd-a2da-4ac9-32a53915c2d0 (at 10.8.3.33@o2ib6) Oct 21 01:32:45 oak-md1-s2 kernel: Lustre: Skipped 221 previous similar messages Oct 21 01:42:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client c5491ebf-d937-4620-8ab8-2cea7b290581 (at 10.8.1.19@o2ib6) reconnecting Oct 21 01:42:18 oak-md1-s2 kernel: Lustre: Skipped 182 previous similar messages Oct 21 01:42:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d296a15e-8154-8527-be08-b302d3e42b22 (at 10.8.6.21@o2ib6) Oct 21 01:42:46 oak-md1-s2 kernel: Lustre: Skipped 224 previous similar messages Oct 21 01:50:15 oak-md1-s2 kernel: LustreError: 4546:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.3.8@o2ib6) returned error from blocking AST (req@ffff8807c0a69b00 x1610634662826960 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880dc6d0b600/0x6c00c4311a8fafda lrc: 4/0,0 mode: PR/PR res: [0x200017a68:0x1edf3:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.8.3.8@o2ib6 remote: 0xaeabb0897d0206cb expref: 19985 pid: 10547 timeout: 8396107874 lvb_type: 0 Oct 21 01:50:15 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.3.8@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 21 01:52:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 6db5d274-ed7f-7baa-2139-5e2a5794c120 (at 10.8.4.26@o2ib6) reconnecting Oct 21 01:52:20 oak-md1-s2 kernel: Lustre: Skipped 383 previous similar messages Oct 21 01:53:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f23f0c89-7fd6-57ad-add6-3f076add893b (at 10.8.4.23@o2ib6) Oct 21 01:53:41 oak-md1-s2 kernel: Lustre: Skipped 348 previous similar messages Oct 21 02:00:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 159577e7-f506-3f52-71e2-15711a269aeb (at 10.8.6.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185bacec00, cur 1540112459 expire 1540112309 last 1540112232 Oct 21 02:06:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 61ce36c6-2d79-34af-219e-ebfe46b8e883 (at 10.8.3.12@o2ib6) reconnecting Oct 21 02:06:20 oak-md1-s2 kernel: Lustre: Skipped 53 previous similar messages Oct 21 02:06:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d036a5bf-7ec8-4c86-6d42-02f20fb9cd2f (at 10.8.3.12@o2ib6) Oct 21 02:06:20 oak-md1-s2 kernel: Lustre: Skipped 40 previous similar messages Oct 21 02:06:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5bbfce3d-2a75-0cd1-35f1-6b75db91823d (at 10.8.4.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881849fd2000, cur 1540112803 expire 1540112653 last 1540112576 Oct 21 02:07:44 oak-md1-s2 kernel: LustreError: 7152:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880b1ec0f050 x1613567756366928/t0(0) o37->b09ab11c-5fa2-4a23-400f-0f3ef3e70d38@10.8.22.8@o2ib6:451/0 lens 568/440 e 0 to 0 dl 1540112871 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 02:14:08 oak-md1-s2 kernel: LustreError: 7144:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff880d39b4f850 x1613497924790448/t0(0) o37->61ce36c6-2d79-34af-219e-ebfe46b8e883@10.8.3.12@o2ib6:79/0 lens 568/440 e 0 to 0 dl 1540113254 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 02:16:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 128b3170-3a01-9359-6a43-e499aeab0560 (at 10.8.4.18@o2ib6) reconnecting Oct 21 02:16:20 oak-md1-s2 kernel: Lustre: Skipped 610 previous similar messages Oct 21 02:16:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3796746c-13cb-1dff-031b-f34216a2f802 (at 10.8.4.18@o2ib6) Oct 21 02:16:20 oak-md1-s2 kernel: Lustre: Skipped 611 previous similar messages Oct 21 02:26:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 3c584b6c-619a-5284-c2d7-34b6f585c3ba (at 10.8.4.31@o2ib6) reconnecting Oct 21 02:26:45 oak-md1-s2 kernel: Lustre: Skipped 250 previous similar messages Oct 21 02:26:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 189897b2-be25-1d7e-aa38-95ade6914239 (at 10.8.4.31@o2ib6) Oct 21 02:26:45 oak-md1-s2 kernel: Lustre: Skipped 250 previous similar messages Oct 21 02:33:28 oak-md1-s2 kernel: LustreError: 4286:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881ea8efbc50 x1613297714789312/t0(0) o37->a370cb3a-832f-adea-ab2a-2935b177b76a@10.8.8.26@o2ib6:521/0 lens 568/440 e 0 to 0 dl 1540114451 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 02:36:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f2b4dd3c-2897-f12d-1195-18fd03df4a02 (at 10.8.6.2@o2ib6) reconnecting Oct 21 02:36:46 oak-md1-s2 kernel: Lustre: Skipped 85 previous similar messages Oct 21 02:36:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d5bbb2f3-d359-d4c1-038d-52b7d34607f5 (at 10.8.6.2@o2ib6) Oct 21 02:36:46 oak-md1-s2 kernel: Lustre: Skipped 85 previous similar messages Oct 21 02:46:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2b6d12da-24cd-ccd4-953a-70a42827225f (at 10.8.3.21@o2ib6) reconnecting Oct 21 02:46:48 oak-md1-s2 kernel: Lustre: Skipped 90 previous similar messages Oct 21 02:46:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dafb6a59-e11b-03ee-af38-b4b95700f3c7 (at 10.8.4.22@o2ib6) Oct 21 02:46:48 oak-md1-s2 kernel: Lustre: Skipped 90 previous similar messages Oct 21 02:56:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 3c584b6c-619a-5284-c2d7-34b6f585c3ba (at 10.8.4.31@o2ib6) reconnecting Oct 21 02:56:56 oak-md1-s2 kernel: Lustre: Skipped 40 previous similar messages Oct 21 02:56:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 189897b2-be25-1d7e-aa38-95ade6914239 (at 10.8.4.31@o2ib6) Oct 21 02:56:56 oak-md1-s2 kernel: Lustre: Skipped 42 previous similar messages Oct 21 03:08:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 06815d56-2a00-16f5-4fb3-2c947067c07f (at 10.8.6.22@o2ib6) reconnecting Oct 21 03:08:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3796746c-13cb-1dff-031b-f34216a2f802 (at 10.8.4.18@o2ib6) Oct 21 03:08:19 oak-md1-s2 kernel: Lustre: Skipped 22 previous similar messages Oct 21 03:08:19 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Oct 21 03:19:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fce99bbe-1488-7f24-bf1b-5b4b70e9df51 (at 10.8.4.23@o2ib6) reconnecting Oct 21 03:19:27 oak-md1-s2 kernel: Lustre: Skipped 19 previous similar messages Oct 21 03:19:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f23f0c89-7fd6-57ad-add6-3f076add893b (at 10.8.4.23@o2ib6) Oct 21 03:19:27 oak-md1-s2 kernel: Lustre: Skipped 20 previous similar messages Oct 21 03:29:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fce99bbe-1488-7f24-bf1b-5b4b70e9df51 (at 10.8.4.23@o2ib6) reconnecting Oct 21 03:29:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3f21a844-2e54-78fb-8e2f-324e13917419 (at 10.8.6.5@o2ib6) Oct 21 03:29:50 oak-md1-s2 kernel: Lustre: Skipped 218 previous similar messages Oct 21 03:29:50 oak-md1-s2 kernel: Lustre: Skipped 219 previous similar messages Oct 21 03:42:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 72716cf8-0d8a-1824-6ea8-2aee66069d58 (at 10.8.3.13@o2ib6) reconnecting Oct 21 03:42:00 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 21 03:42:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 66367ed6-6ef8-36b5-068a-c130922fe990 (at 10.8.3.13@o2ib6) Oct 21 03:42:00 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Oct 21 09:12:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa5541c6-b49f-97f9-7ebd-59eaa5151787 (at 10.12.4.35@o2ib) Oct 21 09:12:21 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 21 09:14:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 99578155-d31b-f5fb-cdca-74b401e806a0 (at 10.12.4.70@o2ib) Oct 21 09:14:37 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 21 09:38:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9cdcd4e-93ba-fba2-9fa9-4da22c874805 (at 10.12.4.71@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b73069c00, cur 1540139929 expire 1540139779 last 1540139702 Oct 21 10:10:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) reconnecting Oct 21 10:10:38 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 21 10:10:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0ac2bcc5-165c-84ca-06ae-bd1a305c4ad4 (at 10.0.2.221@o2ib5) Oct 21 10:10:38 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 21 11:36:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 62e67c48-9e56-ec95-6b34-2d4743010451 (at 10.8.11.33@o2ib6) reconnecting Oct 21 11:36:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2d39e137-e7fc-0b0e-ffa4-c1aa9e95dcc7 (at 10.8.11.33@o2ib6) Oct 21 11:47:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client edfe11aa-0117-e8a3-9ce3-ea2080ae0f62 (at 10.8.4.3@o2ib6) reconnecting Oct 21 11:47:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bf6ed993-7a7e-915b-0abe-6b51a5acb8b8 (at 10.8.4.3@o2ib6) Oct 21 11:51:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 32178a37-a663-b83b-4e79-7dc56bdb6d9d (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817e0f13c00, cur 1540147884 expire 1540147734 last 1540147657 Oct 21 11:54:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac820459-5de4-bcdc-380d-030cf5a49a84 (at 10.12.4.35@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fe1340c00, cur 1540148093 expire 1540147943 last 1540147866 Oct 21 11:56:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa5541c6-b49f-97f9-7ebd-59eaa5151787 (at 10.12.4.35@o2ib) Oct 21 11:59:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f2b4dd3c-2897-f12d-1195-18fd03df4a02 (at 10.8.6.2@o2ib6) reconnecting Oct 21 11:59:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d5bbb2f3-d359-d4c1-038d-52b7d34607f5 (at 10.8.6.2@o2ib6) Oct 21 12:01:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 128b3170-3a01-9359-6a43-e499aeab0560 (at 10.8.4.18@o2ib6) reconnecting Oct 21 12:01:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3796746c-13cb-1dff-031b-f34216a2f802 (at 10.8.4.18@o2ib6) Oct 21 12:02:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 32e72eda-b1e1-e5e5-1063-1fc7a9111f9e (at 10.8.27.6@o2ib6) reconnecting Oct 21 12:02:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 083460d3-0d7b-f761-6e04-bdf3e608886a (at 10.8.27.6@o2ib6) Oct 21 12:02:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client cf600f5c-8ffb-c459-439b-e0ba47aa6951 (at 10.8.27.12@o2ib6) reconnecting Oct 21 12:04:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f6e43750-0ed4-cc40-a577-c756a5b956c0 (at 10.8.6.1@o2ib6) reconnecting Oct 21 12:04:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 21 12:04:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c1afa067-c9f8-ff8a-3bfc-32df39b983c5 (at 10.8.6.1@o2ib6) Oct 21 12:04:02 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 21 12:06:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 1e5be92f-a52e-a471-c44c-e78982d16e87 (at 10.8.4.29@o2ib6) reconnecting Oct 21 12:06:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6e7dbe8-678f-5bec-58d6-4a582a6d69c2 (at 10.8.4.29@o2ib6) Oct 21 12:08:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 15c45df1-17c8-57b3-7d43-907484f3c1b3 (at 10.8.27.3@o2ib6) reconnecting Oct 21 12:08:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 24f4fd6e-d5fd-d892-fb49-ac284f050442 (at 10.8.27.3@o2ib6) Oct 21 12:12:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fce99bbe-1488-7f24-bf1b-5b4b70e9df51 (at 10.8.4.23@o2ib6) reconnecting Oct 21 12:12:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f23f0c89-7fd6-57ad-add6-3f076add893b (at 10.8.4.23@o2ib6) Oct 21 12:15:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a2799c92-ec6d-092f-255a-aceb9773f13b (at 10.8.27.4@o2ib6) reconnecting Oct 21 12:15:29 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 21 12:15:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b82e363f-2682-19f8-c12e-347146acdfda (at 10.8.27.4@o2ib6) Oct 21 12:15:29 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 21 12:17:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client b09ab11c-5fa2-4a23-400f-0f3ef3e70d38 (at 10.8.22.8@o2ib6) reconnecting Oct 21 12:17:39 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 21 12:19:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43a05b34-4ef7-6ca4-916f-2782eebf9a37 (at 10.8.4.26@o2ib6) Oct 21 12:19:54 oak-md1-s2 kernel: Lustre: Skipped 17 previous similar messages Oct 21 12:22:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 84a11ee2-b773-4af1-2d34-8c377808e1bf (at 10.8.31.2@o2ib6) reconnecting Oct 21 12:22:52 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Oct 21 12:29:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b4fe4e8a-a292-d39b-efe4-3c357f0f782d (at 10.8.3.36@o2ib6) Oct 21 12:29:46 oak-md1-s2 kernel: Lustre: Skipped 42 previous similar messages Oct 21 12:31:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 426aaed5-1677-624c-ab0b-ff023557cca9 (at 10.8.3.24@o2ib6) reconnecting Oct 21 12:31:29 oak-md1-s2 kernel: Lustre: Skipped 39 previous similar messages Oct 21 12:40:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to eaee074c-c42a-1b0d-d24f-6a4e0ab844fd (at 10.8.3.10@o2ib6) Oct 21 12:40:14 oak-md1-s2 kernel: Lustre: Skipped 53 previous similar messages Oct 21 12:41:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f0b5d0b6-07ca-2779-b47b-2586de460798 (at 10.8.3.32@o2ib6) reconnecting Oct 21 12:41:41 oak-md1-s2 kernel: Lustre: Skipped 45 previous similar messages Oct 21 12:50:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1dd0d451-6fc3-8a05-bce3-81b0ce7653a7 (at 10.8.2.1@o2ib6) Oct 21 12:50:17 oak-md1-s2 kernel: Lustre: Skipped 24 previous similar messages Oct 21 12:52:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 94971fd6-fcd9-64a8-8c33-646cc6673b4b (at 10.8.30.36@o2ib6) reconnecting Oct 21 12:52:09 oak-md1-s2 kernel: Lustre: Skipped 26 previous similar messages Oct 21 13:00:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43a05b34-4ef7-6ca4-916f-2782eebf9a37 (at 10.8.4.26@o2ib6) Oct 21 13:00:18 oak-md1-s2 kernel: Lustre: Skipped 35 previous similar messages Oct 21 13:02:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 5a3105ed-59c5-5f93-8684-454f88a82d95 (at 10.8.27.1@o2ib6) reconnecting Oct 21 13:02:21 oak-md1-s2 kernel: Lustre: Skipped 66 previous similar messages Oct 21 13:02:37 oak-md1-s2 kernel: LustreError: 11707:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881daf939850 x1613323092741712/t0(0) o37->4cca9c4a-8075-ad50-0b8e-ca13c6e4ad1f@10.8.3.3@o2ib6:484/0 lens 568/440 e 0 to 0 dl 1540152164 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 13:07:49 oak-md1-s2 kernel: LustreError: 3873:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8802bed60600 x1613947357661504/t0(0) o37->9f6c45ea-e2a2-00f5-6d54-f82ae66ccb87@10.8.3.4@o2ib6:79/0 lens 568/440 e 0 to 0 dl 1540152514 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 13:08:18 oak-md1-s2 kernel: LustreError: 4275:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8804749c8f00 x1613285504760304/t0(0) o37->7767708c-e5c7-9720-c135-402e69744c72@10.8.3.18@o2ib6:109/0 lens 568/440 e 0 to 0 dl 1540152544 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 13:12:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 15253068-2c1a-9e16-a713-fd46ee5ff2b1 (at 10.8.3.22@o2ib6) reconnecting Oct 21 13:12:49 oak-md1-s2 kernel: Lustre: Skipped 118 previous similar messages Oct 21 13:12:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to cb9e603f-bb8f-11d1-6162-4080c55179f0 (at 10.8.3.22@o2ib6) Oct 21 13:12:49 oak-md1-s2 kernel: Lustre: Skipped 153 previous similar messages Oct 21 13:22:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client e2a63b25-c0db-b110-ebcb-fe54ac2c02fc (at 10.8.2.4@o2ib6) reconnecting Oct 21 13:22:57 oak-md1-s2 kernel: Lustre: Skipped 82 previous similar messages Oct 21 13:22:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to feeee0a9-94d1-55ba-1a7d-affde36dfe4f (at 10.8.2.4@o2ib6) Oct 21 13:22:58 oak-md1-s2 kernel: Lustre: Skipped 82 previous similar messages Oct 21 13:26:54 oak-md1-s2 kernel: LustreError: 4275:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88015ce52700 x1613325161099312/t0(0) o37->f53687fe-917d-4f1d-f88d-4e2752c8aa5c@10.8.6.23@o2ib6:437/0 lens 568/440 e 0 to 0 dl 1540153627 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 13:33:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 54bfab66-92c2-1e7d-2978-218231da25e0 (at 10.8.0.65@o2ib6) reconnecting Oct 21 13:33:04 oak-md1-s2 kernel: Lustre: Skipped 182 previous similar messages Oct 21 13:33:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ef02aec4-f626-b40d-d404-973e88e07efe (at 10.8.0.65@o2ib6) Oct 21 13:33:04 oak-md1-s2 kernel: Lustre: Skipped 182 previous similar messages Oct 21 13:39:02 oak-md1-s2 kernel: LustreError: 181383:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff881a042a1b00 x1613400534522704/t0(0) o37->3c584b6c-619a-5284-c2d7-34b6f585c3ba@10.8.4.31@o2ib6:404/0 lens 568/440 e 0 to 0 dl 1540154349 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 13:43:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 6db5d274-ed7f-7baa-2139-5e2a5794c120 (at 10.8.4.26@o2ib6) reconnecting Oct 21 13:43:29 oak-md1-s2 kernel: Lustre: Skipped 272 previous similar messages Oct 21 13:43:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 43a05b34-4ef7-6ca4-916f-2782eebf9a37 (at 10.8.4.26@o2ib6) Oct 21 13:43:29 oak-md1-s2 kernel: Lustre: Skipped 272 previous similar messages Oct 21 13:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 9f74b84c-30e2-127c-17ac-84a4c160a3d9 (at 10.8.6.3@o2ib6) reconnecting Oct 21 13:53:45 oak-md1-s2 kernel: Lustre: Skipped 64 previous similar messages Oct 21 13:53:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3f83601e-2f19-ecd1-e8a4-e59a6070c6d5 (at 10.8.6.3@o2ib6) Oct 21 13:53:45 oak-md1-s2 kernel: Lustre: Skipped 64 previous similar messages Oct 21 13:57:17 oak-md1-s2 kernel: LustreError: 4245:0:(ldlm_lib.c:3247:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8810949c2450 x1613487056874320/t0(0) o37->f2b4dd3c-2897-f12d-1195-18fd03df4a02@10.8.6.2@o2ib6:743/0 lens 568/440 e 0 to 0 dl 1540155443 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 14:07:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 94971fd6-fcd9-64a8-8c33-646cc6673b4b (at 10.8.30.36@o2ib6) reconnecting Oct 21 14:07:00 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Oct 21 14:07:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85d90c94-e070-68cc-c88b-2b519e41b039 (at 10.8.30.36@o2ib6) Oct 21 14:07:00 oak-md1-s2 kernel: Lustre: Skipped 12 previous similar messages Oct 21 14:21:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 94971fd6-fcd9-64a8-8c33-646cc6673b4b (at 10.8.30.36@o2ib6) reconnecting Oct 21 14:21:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 21 14:21:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85d90c94-e070-68cc-c88b-2b519e41b039 (at 10.8.30.36@o2ib6) Oct 21 14:21:29 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 21 16:18:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 21 16:18:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c5b5e61a-ad81-6b80-8aae-d2e0903a0189 (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e71c9000, cur 1540163895 expire 1540163745 last 1540163668 Oct 21 16:23:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 15c45df1-17c8-57b3-7d43-907484f3c1b3 (at 10.8.27.3@o2ib6) reconnecting Oct 21 16:23:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 24f4fd6e-d5fd-d892-fb49-ac284f050442 (at 10.8.27.3@o2ib6) Oct 21 16:24:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client bc055b94-aae6-3761-985b-d9776df0bc31 (at 10.8.20.18@o2ib6) reconnecting Oct 21 16:27:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 42b6a600-87dd-122b-84db-957e0c49ee6e (at 10.8.17.2@o2ib6) reconnecting Oct 21 16:27:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bee5548d-becb-3e18-2501-11666f7a6bf6 (at 10.8.17.2@o2ib6) Oct 21 16:27:35 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 21 16:28:12 oak-md1-s2 kernel: LustreError: 7167:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff88101cff0850 x1613433217970032/t0(0) o37->d1db00f4-9c8e-f059-7cc5-06d4865bfc7c@10.8.20.16@o2ib6:20/0 lens 568/440 e 0 to 0 dl 1540164535 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 16:34:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client d618276b-cb16-9db8-2743-08324391bdfe (at 10.8.22.33@o2ib6) reconnecting Oct 21 16:34:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 21 16:34:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a51c2751-2210-5632-0d46-a3c2ad9d9c6e (at 10.8.22.33@o2ib6) Oct 21 16:34:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 21 16:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 67b39aa9-36a7-4d37-1b3d-d61a27269314 (at 10.8.27.15@o2ib6) reconnecting Oct 21 16:44:49 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Oct 21 16:44:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e7f0c4c0-757e-2d84-6f1c-f68c4d15d371 (at 10.8.27.15@o2ib6) Oct 21 16:44:49 oak-md1-s2 kernel: Lustre: Skipped 23 previous similar messages Oct 21 16:55:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2af79cbf-42f1-30d4-c1c0-3e2bae0dbb38 (at 10.8.24.2@o2ib6) reconnecting Oct 21 16:55:19 oak-md1-s2 kernel: Lustre: Skipped 24 previous similar messages Oct 21 16:55:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 12db1ac1-fb55-0efe-45d3-04131d59774b (at 10.8.24.2@o2ib6) Oct 21 16:55:19 oak-md1-s2 kernel: Lustre: Skipped 24 previous similar messages Oct 21 17:14:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a2799c92-ec6d-092f-255a-aceb9773f13b (at 10.8.27.4@o2ib6) reconnecting Oct 21 17:14:47 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 21 17:14:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b82e363f-2682-19f8-c12e-347146acdfda (at 10.8.27.4@o2ib6) Oct 21 17:14:47 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 21 17:32:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 4b036d20-6f05-a19d-b47e-da45f6ebe5e9 (at 10.8.18.26@o2ib6) reconnecting Oct 21 17:32:18 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 21 17:32:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b6b47c6a-1a99-9cb2-910d-58c513f764e9 (at 10.8.18.26@o2ib6) Oct 21 17:32:18 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 21 17:42:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 7836d117-41d3-a5d5-761c-ef99b0ab7549 (at 10.8.7.31@o2ib6) reconnecting Oct 21 17:42:19 oak-md1-s2 kernel: Lustre: Skipped 19 previous similar messages Oct 21 17:42:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 455e2c1f-697b-fb8d-971b-2dbeb1ba0d2f (at 10.8.7.31@o2ib6) Oct 21 17:42:19 oak-md1-s2 kernel: Lustre: Skipped 19 previous similar messages Oct 21 17:53:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 71a2c776-ace1-3ede-2f50-6acd5dbd485a (at 10.8.1.18@o2ib6) reconnecting Oct 21 17:53:21 oak-md1-s2 kernel: Lustre: Skipped 36 previous similar messages Oct 21 17:53:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2455eb3f-12df-a8cc-df9f-a0917ba1883c (at 10.8.1.18@o2ib6) Oct 21 17:53:21 oak-md1-s2 kernel: Lustre: Skipped 36 previous similar messages Oct 21 18:03:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 32e72eda-b1e1-e5e5-1063-1fc7a9111f9e (at 10.8.27.6@o2ib6) reconnecting Oct 21 18:03:23 oak-md1-s2 kernel: Lustre: Skipped 41 previous similar messages Oct 21 18:03:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 083460d3-0d7b-f761-6e04-bdf3e608886a (at 10.8.27.6@o2ib6) Oct 21 18:03:23 oak-md1-s2 kernel: Lustre: Skipped 41 previous similar messages Oct 21 18:12:57 oak-md1-s2 kernel: LustreError: 4271:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff880cf476c450 x1613526567673920/t0(0) o37->32e72eda-b1e1-e5e5-1063-1fc7a9111f9e@10.8.27.6@o2ib6:228/0 lens 568/440 e 0 to 0 dl 1540170783 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 18:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 6cde2bb8-e9d8-b93b-64e3-66dea9fbb0d9 (at 10.8.17.4@o2ib6) reconnecting Oct 21 18:13:23 oak-md1-s2 kernel: Lustre: Skipped 627 previous similar messages Oct 21 18:13:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 56028650-e241-082e-4b54-2066bbd145d9 (at 10.8.17.4@o2ib6) Oct 21 18:13:23 oak-md1-s2 kernel: Lustre: Skipped 627 previous similar messages Oct 21 18:22:21 oak-md1-s2 kernel: LustreError: 4259:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881634730600 x1613514152723248/t0(0) o37->eeb4b74f-549f-d327-e480-81ac5f6e9780@10.8.18.4@o2ib6:37/0 lens 568/440 e 0 to 0 dl 1540171347 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 18:23:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client a9f2eaf4-67fc-d2cc-d124-abd08beaa8d3 (at 10.8.18.2@o2ib6) reconnecting Oct 21 18:23:26 oak-md1-s2 kernel: Lustre: Skipped 1155 previous similar messages Oct 21 18:23:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 327e59a5-68a2-1830-e9e7-199c061eead4 (at 10.8.18.2@o2ib6) Oct 21 18:23:26 oak-md1-s2 kernel: Lustre: Skipped 1155 previous similar messages Oct 21 18:24:05 oak-md1-s2 kernel: LustreError: 181523:0:(ldlm_lib.c:3197:target_bulk_io()) @@@ bulk READ failed: rc -107 req@ffff881edd252100 x1613513874823568/t0(0) o37->0cc30097-9c10-00eb-665b-baad25ade550@10.8.18.3@o2ib6:141/0 lens 568/440 e 0 to 0 dl 1540171451 ref 1 fl Interpret:/0/0 rc 0/0 Oct 21 18:35:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client f4c29ecb-cb92-3117-56a0-6cab000f2052 (at 10.8.8.4@o2ib6) reconnecting Oct 21 18:35:28 oak-md1-s2 kernel: Lustre: Skipped 170 previous similar messages Oct 21 18:35:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 49c29614-7d2c-7e6e-e448-f77deed0dbbe (at 10.8.8.4@o2ib6) Oct 21 18:35:28 oak-md1-s2 kernel: Lustre: Skipped 170 previous similar messages Oct 21 20:34:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39356fa0-76d3-d56a-9a46-e95580e3b0f3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880004432400, cur 1540179280 expire 1540179130 last 1540179053 Oct 21 20:37:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 21 20:37:46 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 21 20:46:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 21 20:47:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dc424902-f77c-d557-d6b6-77a2eb3355a4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dc080d400, cur 1540180043 expire 1540179893 last 1540179816 Oct 21 20:49:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 21 20:50:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e9ac0242-7643-8b7d-d4cc-e695c90516b3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ff6156c00, cur 1540180244 expire 1540180094 last 1540180017 Oct 21 21:05:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 36d6b961-76ea-7123-9228-c797197e2961 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88080dbc1800, cur 1540181155 expire 1540181005 last 1540180928 Oct 21 22:10:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1542e0c4-1414-0d63-8bf3-7d78d534d1f3 (at 10.8.14.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6d1d8400, cur 1540185032 expire 1540184882 last 1540184805 Oct 21 23:31:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0c553792-48ab-c2d8-c32a-e95c7c1025dc (at 10.9.105.66@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe534a400, cur 1540189917 expire 1540189767 last 1540189690 Oct 22 05:19:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client cd6bd1b3-1b72-bed3-d8c7-500c34cf7a43 (at 10.8.7.19@o2ib6) reconnecting Oct 22 05:19:37 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 22 05:19:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b9a3194e-19c3-f27c-34cc-b6dd41be449b (at 10.8.7.19@o2ib6) Oct 22 07:44:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 22 07:46:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9c3cb063-6a91-bb9f-e294-3feca9523657 (at 10.8.2.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880510a9ac00, cur 1540219585 expire 1540219435 last 1540219358 Oct 22 07:51:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Oct 22 08:02:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ef5987d3-a3e3-ac25-d801-021c3197b71d (at 10.8.2.26@o2ib6) Oct 22 08:02:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c86a420f-c781-b7d5-4291-9e94489b056b (at 10.8.2.16@o2ib6) Oct 22 08:02:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 08:02:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1b79d1e8-945a-b9cf-fb85-e5f9e3598a24 (at 10.8.2.28@o2ib6) Oct 22 08:02:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 08:02:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1b5bd045-fc90-c679-8b71-cc5361bcdd13 (at 10.8.2.25@o2ib6) Oct 22 08:02:55 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 08:03:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 796dc795-c10f-42a6-774f-ae9c0954f5e6 (at 10.8.2.17@o2ib6) Oct 22 08:03:42 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 22 08:04:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0fc3c314-e03c-b178-7e24-1ae4ccb8f808 (at 10.8.4.1@o2ib6) Oct 22 08:07:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Oct 22 08:07:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 08:09:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e4c62829-1422-3f51-7b3e-6add67ab8070 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d25400, cur 1540220983 expire 1540220833 last 1540220756 Oct 22 08:09:43 oak-md1-s2 kernel: Lustre: Skipped 17 previous similar messages Oct 22 08:19:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bccf629c-900e-c4a9-171a-d70129138265 (at 10.8.14.1@o2ib6) Oct 22 08:19:11 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 22 08:22:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3dacaa73-1e9b-1c9e-ca6e-450bf376bf78 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d1a24b400, cur 1540221723 expire 1540221573 last 1540221496 Oct 22 08:31:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1c1dbb6d-dcc9-954e-6d48-f2c4d659969d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f565cdc00, cur 1540222260 expire 1540222110 last 1540222033 Oct 22 08:33:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 08:33:25 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 08:42:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c181bf0f-5065-3967-9f97-fb6eff230c4d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185c29ac00, cur 1540222966 expire 1540222816 last 1540222739 Oct 22 08:44:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4778239f-e7d8-bfb7-af61-b1ac83e1f473 (at 10.8.20.15@o2ib6) in 211 seconds. I think it's dead, and I am evicting it. exp ffff881ffae97800, cur 1540223042 expire 1540222892 last 1540222831 Oct 22 08:46:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 08:46:07 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 22 08:47:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3927f7de-1bf9-cb87-9f69-d74e36725486 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d2be18c00, cur 1540223241 expire 1540223091 last 1540223014 Oct 22 08:53:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 11e15438-7bb9-6f9d-a4cf-ccb62ace77be (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881856ff0800, cur 1540223595 expire 1540223445 last 1540223368 Oct 22 08:54:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a07740fc-edd6-fc61-ad5c-b2e9c6ac96e5 (at 10.8.18.35@o2ib6) in 163 seconds. I think it's dead, and I am evicting it. exp ffff881c9e73d400, cur 1540223671 expire 1540223521 last 1540223508 Oct 22 09:01:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 09:01:49 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 22 09:01:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3433bb9f-1671-1d90-6b99-571abcd41337 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881486023400, cur 1540224116 expire 1540223966 last 1540223889 Oct 22 09:10:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f526793-a37e-1695-5521-f3e69e9b1339 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031dc76000, cur 1540224611 expire 1540224461 last 1540224384 Oct 22 09:10:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 09:19:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 09:19:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 09:23:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f34eacd2-9ac2-4406-c9da-434f86cfaf75 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880106780800, cur 1540225384 expire 1540225234 last 1540225157 Oct 22 09:29:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 22 09:29:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2197da50-93bb-4a57-a3a4-9bafe617b1be (at 10.0.2.3@o2ib5) Oct 22 09:29:27 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 09:29:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 22 09:30:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 22 09:30:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 07e016c3-47f3-5286-0731-e374975ffaf9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881816ee0c00, cur 1540225845 expire 1540225695 last 1540225618 Oct 22 09:31:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client fc36c3db-5207-2dff-a9c3-3e2dab439f91 (at 10.0.2.3@o2ib5) reconnecting Oct 22 09:31:16 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 09:34:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 36307c1d-a0a0-a9b6-a031-868052ddeccf (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800264a1800, cur 1540226084 expire 1540225934 last 1540225857 Oct 22 09:40:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 790579ec-3058-a557-26cb-b27effa404dc (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b7702400, cur 1540226412 expire 1540226262 last 1540226185 Oct 22 09:40:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 09:40:34 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 22 09:53:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 09:54:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ae1e3342-0038-0cde-0328-322ada3911af (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d0e230000, cur 1540227261 expire 1540227111 last 1540227034 Oct 22 10:01:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 63c7c392-120c-b63b-c59f-ea2d56b2d308 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b643b400, cur 1540227668 expire 1540227518 last 1540227441 Oct 22 10:10:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4632229f-7fee-31d1-b85b-69776b7afa30 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880003c55400, cur 1540228204 expire 1540228054 last 1540227977 Oct 22 10:11:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 10:11:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 10:15:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 73c2c5a2-9900-4265-a728-66e15c734036 (at 10.8.2.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814f06d7800, cur 1540228546 expire 1540228396 last 1540228319 Oct 22 10:20:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6b31bb19-966d-1a17-a173-d8ba5b7c43a3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881db00d2c00, cur 1540228837 expire 1540228687 last 1540228610 Oct 22 10:22:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 22 10:22:02 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 10:27:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d28d2f41-a342-3816-5fff-8cb39475c967 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e42848400, cur 1540229275 expire 1540229125 last 1540229048 Oct 22 10:40:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 10:40:28 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 22 10:47:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b816f86f-2cc4-ca4c-97b8-e83836a307de (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e70792800, cur 1540230431 expire 1540230281 last 1540230204 Oct 22 10:47:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 10:52:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 10:52:12 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 22 10:53:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c585b810-b8da-027b-f9d8-a7756ca7f82a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b1ec0c000, cur 1540230781 expire 1540230631 last 1540230554 Oct 22 11:11:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 11:11:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2a63b25-c0db-b110-ebcb-fe54ac2c02fc (at 10.8.2.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881812762000, cur 1540231879 expire 1540231729 last 1540231652 Oct 22 11:27:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to feeee0a9-94d1-55ba-1a7d-affde36dfe4f (at 10.8.2.4@o2ib6) Oct 22 11:32:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ae8797e-f325-d958-0461-812d91ea647d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8816ca118c00, cur 1540233121 expire 1540232971 last 1540232894 Oct 22 11:32:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 22 12:04:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 27154ff3-4029-838b-7dc2-6bcc62c010f4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819fff9c400, cur 1540235074 expire 1540234924 last 1540234847 Oct 22 12:06:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 12:06:43 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 22 12:12:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 82066c67-f97b-798a-15dc-f4d9f166b2bd (at 10.12.4.35@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880197a56400, cur 1540235520 expire 1540235370 last 1540235293 Oct 22 12:15:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 632bc196-4ec5-ab50-ff9e-38b5025011f8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881088df6c00, cur 1540235701 expire 1540235551 last 1540235474 Oct 22 12:16:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 15cfe488-bee5-e58b-9db4-b66522df73f0 (at 10.8.20.15@o2ib6) in 199 seconds. I think it's dead, and I am evicting it. exp ffff88127d2e6400, cur 1540235777 expire 1540235627 last 1540235578 Oct 22 12:19:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 12:27:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client efe85f42-c94a-b0ad-397f-78e6e0696563 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d72e30800, cur 1540236429 expire 1540236279 last 1540236202 Oct 22 12:27:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 12:29:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 12:31:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b95be6dd-21ee-5d35-ac98-c1e2fe4a66b9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d72e30c00, cur 1540236665 expire 1540236515 last 1540236438 Oct 22 12:37:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 410d5103-85b4-3152-5c39-a0c4b0ca70d2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803dd665800, cur 1540237039 expire 1540236889 last 1540236812 Oct 22 12:40:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 12:53:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 069020d0-2d38-d371-2c4a-363d71f814f7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815972e2000, cur 1540238037 expire 1540237887 last 1540237810 Oct 22 12:54:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 13:00:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 13:01:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e6dbeaec-788a-f428-b109-b24b1b3395db (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881558a5a400, cur 1540238506 expire 1540238356 last 1540238279 Oct 22 13:01:46 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 13:07:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 13:07:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f60e9f60-b0ab-024f-6224-249aaa84f8ed (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88185c662000, cur 1540238874 expire 1540238724 last 1540238647 Oct 22 13:09:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2c5f7722-4fad-d38d-f918-0a4dcb48f218 (at 10.8.2.7@o2ib6) in 188 seconds. I think it's dead, and I am evicting it. exp ffff881dac930c00, cur 1540238950 expire 1540238800 last 1540238762 Oct 22 13:09:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 13:13:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 13:14:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 12872ae0-ec7f-ed2a-611b-a5e86abc33cf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88170039ec00, cur 1540239249 expire 1540239099 last 1540239022 Oct 22 13:19:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5bf085c4-7698-cb73-d91c-d388aeb8271a (at 10.8.2.14@o2ib6) Oct 22 13:24:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 47e4fec9-3f70-966d-e91f-dba9c2d7ca48 (at 10.8.2.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e71ce800, cur 1540239864 expire 1540239714 last 1540239637 Oct 22 13:25:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8ff77b8d-b85d-6077-7091-ce9b634eda8d (at 10.8.2.7@o2ib6) Oct 22 13:33:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4f5c0c36-67ee-7364-1d27-3414a7a20d69 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881898682000, cur 1540240380 expire 1540240230 last 1540240153 Oct 22 13:38:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8dbd6139-2097-ab63-d9a3-e6c5d4e12db0 (at 10.9.114.6@o2ib4) Oct 22 13:39:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a5f20b24-b114-84ae-8a9d-c54e7d97545c (at 10.9.114.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e38758c00, cur 1540240776 expire 1540240626 last 1540240549 Oct 22 13:50:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to dd37ab47-2674-29f4-6a97-a17dda5b9850 (at 10.8.2.13@o2ib6) Oct 22 13:53:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 22 14:04:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 22 14:06:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 144dc5c4-0647-3e2d-9452-9e7f3160c5cc (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881540796c00, cur 1540242375 expire 1540242225 last 1540242148 Oct 22 14:21:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c0db7b2-7ee1-547a-c66d-1241aac679ad (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe5a59800, cur 1540243278 expire 1540243128 last 1540243051 Oct 22 14:43:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 40eafa09-5a9c-f747-1408-ca89a9f4c07a (at 10.8.8.9@o2ib6) reconnecting Oct 22 14:43:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a2bcffeb-24ac-3558-3eca-f02626d5362e (at 10.8.8.9@o2ib6) Oct 22 14:43:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 22 15:05:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2aeb3b76-2464-bb79-ae4d-b6b7dc2b73c4 (at 10.12.4.22@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88010988fc00, cur 1540245939 expire 1540245789 last 1540245712 Oct 22 15:11:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 83328f9b-784f-03e7-c891-3fae2455dff3 (at 10.12.4.25@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e3c25000, cur 1540246317 expire 1540246167 last 1540246090 Oct 22 15:13:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d625dd8c-9e37-f914-9817-44f7c127baba (at 10.12.4.26@o2ib) in 155 seconds. I think it's dead, and I am evicting it. exp ffff88096d3f4000, cur 1540246393 expire 1540246243 last 1540246238 Oct 22 15:21:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:25:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e69ea91c-174a-d387-2406-d99fa9d7b356 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f2f2abc00, cur 1540247131 expire 1540246981 last 1540246904 Oct 22 15:26:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:29:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e0ef77f-5d69-1b13-37a0-b5674f4d7f3b (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf36d4400, cur 1540247355 expire 1540247205 last 1540247128 Oct 22 15:30:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:34:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 94e4140c-f0d8-0cc7-929a-05adb928dc30 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881800347400, cur 1540247657 expire 1540247507 last 1540247430 Oct 22 15:34:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 15:34:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:38:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:38:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a39d3931-0620-b0c8-b3c2-a3e0959976ba (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0d5000, cur 1540247923 expire 1540247773 last 1540247696 Oct 22 15:42:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c1dc5096-3dd1-71cd-06d1-db3c016132ae (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf93e000, cur 1540248189 expire 1540248039 last 1540247962 Oct 22 15:47:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:47:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e0d3262e-1c62-984d-6233-c927fbd41210 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e6623800, cur 1540248458 expire 1540248308 last 1540248231 Oct 22 15:48:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 22 15:48:56 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 22 15:54:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to de791222-49d9-469f-9ca1-ac7d516c16ce (at 10.8.15.10@o2ib6) Oct 22 15:56:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce69d950-68eb-0ee6-e2fa-c2e4514f89e2 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880af70d2800, cur 1540248989 expire 1540248839 last 1540248762 Oct 22 15:57:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0009f73f-9e24-0f9c-ef0d-11a8386eb31b (at 10.8.9.5@o2ib6) in 179 seconds. I think it's dead, and I am evicting it. exp ffff881dc317dc00, cur 1540249065 expire 1540248915 last 1540248886 Oct 22 15:58:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 15:58:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 16:02:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9ae9ac9a-f41f-fc11-bce2-acf7180a16b1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880779d3e000, cur 1540249360 expire 1540249210 last 1540249133 Oct 22 16:02:40 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 16:18:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client edfe11aa-0117-e8a3-9ce3-ea2080ae0f62 (at 10.8.4.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c43be0800, cur 1540250317 expire 1540250167 last 1540250090 Oct 22 16:32:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bf6ed993-7a7e-915b-0abe-6b51a5acb8b8 (at 10.8.4.3@o2ib6) Oct 22 16:32:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 22 17:06:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 979710f8-a1ba-5039-eda8-0c4ac2e2b295 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93135000, cur 1540253189 expire 1540253039 last 1540252962 Oct 22 17:19:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 167059a8-887d-f282-1560-0fbcdd188dab (at 10.8.13.23@o2ib6) Oct 22 17:22:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0d0a4dda-d6b1-2a80-b8d4-71c4511758a3 (at 10.9.105.66@o2ib4) Oct 22 17:24:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f2874060-6bfb-c6c5-c8a8-60d8ca27c3be (at 10.8.6.7@o2ib6) Oct 22 17:31:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9d784eb0-c8cd-910d-f6e1-3baaa62c65d9 (at 10.8.11.10@o2ib6) Oct 22 19:52:37 oak-md1-s2 kernel: LustreError: 4302:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.114.6@o2ib4) failed to reply to blocking AST (req@ffff881cbbf32700 x1610635260733328 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881e22ab4200/0x6c00c44438f27f9b lrc: 4/0,0 mode: PR/PR res: [0x200018370:0x2105:0x0].0x0 bits 0x20 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.114.6@o2ib4 remote: 0x23a4e183ddf064c4 expref: 6489 pid: 4302 timeout: 8548250557 lvb_type: 0 Oct 22 19:52:37 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.114.6@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Oct 22 19:52:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8dbd6139-2097-ab63-d9a3-e6c5d4e12db0 (at 10.9.114.6@o2ib4) Oct 22 19:59:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 22 19:59:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2544e292-de72-86d0-deee-daad769ee806 (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805acd2e400, cur 1540263569 expire 1540263419 last 1540263342 Oct 22 20:00:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 55129298-c954-8b6b-b8cb-e7d6017315c7 (at 10.8.20.15@o2ib6) in 211 seconds. I think it's dead, and I am evicting it. exp ffff88185c29ac00, cur 1540263645 expire 1540263495 last 1540263434 Oct 22 20:03:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 22 20:07:15 oak-md1-s2 kernel: Lustre: 10592:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540264028/real 1540264028] req@ffff8810b7f46300 x1610635261083344/t0(0) o104->oak-MDT0000@10.9.105.43@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540264035 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 22 20:07:22 oak-md1-s2 kernel: Lustre: 10592:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540264035/real 1540264035] req@ffff8810b7f46300 x1610635261083344/t0(0) o104->oak-MDT0000@10.9.105.43@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540264042 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 22 20:07:27 oak-md1-s2 kernel: Lustre: 10671:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540264040/real 1540264040] req@ffff881ee60c8000 x1610635261088032/t0(0) o104->oak-MDT0000@10.9.105.43@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540264047 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 22 20:07:29 oak-md1-s2 kernel: Lustre: 10592:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540264042/real 1540264042] req@ffff8810b7f46300 x1610635261083344/t0(0) o104->oak-MDT0000@10.9.105.43@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540264049 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 22 20:07:34 oak-md1-s2 kernel: Lustre: 10671:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540264047/real 1540264047] req@ffff881ee60c8000 x1610635261088032/t0(0) o104->oak-MDT0000@10.9.105.43@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540264054 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 22 20:07:44 oak-md1-s2 kernel: Lustre: 10592:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540264056/real 1540264056] req@ffff8810b7f46300 x1610635261083344/t0(0) o104->oak-MDT0000@10.9.105.43@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540264063 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 22 20:07:44 oak-md1-s2 kernel: Lustre: 10592:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 22 20:07:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 7673bf88-807c-775e-e212-9e40518c7cc6 (at 10.9.105.43@o2ib4) Oct 22 20:07:55 oak-md1-s2 kernel: LustreError: 10671:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.43@o2ib4) returned error from blocking AST (req@ffff881ee60c8000 x1610635261088032 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff880e3e977600/0x6c00c4441dc70998 lrc: 4/0,0 mode: PR/PR res: [0x2000181e8:0xc0be:0x0].0x0 bits 0x13 rrc: 9 type: IBT flags: 0x60200400000020 nid: 10.9.105.43@o2ib4 remote: 0xa2b5d01d161084ad expref: 9437 pid: 10533 timeout: 8549168406 lvb_type: 0 Oct 22 20:07:55 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.43@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Oct 22 20:07:55 oak-md1-s2 kernel: LustreError: 10592:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8810b7f47800 x1610635261099920/t0(0) o104->oak-MDT0000@10.9.105.43@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 22 20:07:55 oak-md1-s2 kernel: LustreError: 10592:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 4 previous similar messages Oct 22 20:51:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 22 20:51:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fea96f59-bc14-6daf-ac33-9804c365e5c2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e61522400, cur 1540266715 expire 1540266565 last 1540266488 Oct 22 21:37:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ddbcc941-6d4c-b53c-79b0-d8313ea33ab0 (at 10.8.26.32@o2ib6) reconnecting Oct 22 21:37:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 163f9150-1d1f-91c2-9c07-e4bd9f0a0f81 (at 10.8.26.32@o2ib6) Oct 23 00:48:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 58005748-6d76-a67a-6907-61aa894e84bd (at 10.12.4.84@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88099725f800, cur 1540280892 expire 1540280742 last 1540280665 Oct 23 01:50:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 933a9db7-3b4c-bb7e-e4e4-ca565688867b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805feaff000, cur 1540284611 expire 1540284461 last 1540284384 Oct 23 03:00:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 03:01:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3c621256-1424-8c61-cf21-3ca897b7bd57 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880761e33400, cur 1540288880 expire 1540288730 last 1540288653 Oct 23 04:28:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 04:31:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4981ae00-107f-d46d-1296-933e86c448f1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880adae76400, cur 1540294309 expire 1540294159 last 1540294082 Oct 23 04:31:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 04:37:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 07e5f5da-a677-8f42-54f2-80a6c796acbd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b7978dc00, cur 1540294640 expire 1540294490 last 1540294413 Oct 23 04:38:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 04:43:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 04:44:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99315cca-12e0-626f-fb95-1a56961bfcae (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881ffae90400, cur 1540295044 expire 1540294894 last 1540294817 Oct 23 04:49:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7c14013e-f008-9081-6244-63e7d814a711 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff882005c30800, cur 1540295367 expire 1540295217 last 1540295140 Oct 23 04:50:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 04:56:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49998772-a32a-c237-6cf5-759b5a1f0cb0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88003d7ca000, cur 1540295777 expire 1540295627 last 1540295550 Oct 23 04:56:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 05:02:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 05:02:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 035db764-5d22-2c4e-0621-0194eb30bdf7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880814647c00, cur 1540296132 expire 1540295982 last 1540295905 Oct 23 05:07:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3d62cc71-bd47-5fef-6a9d-3a3d209a785b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803dd666400, cur 1540296455 expire 1540296305 last 1540296228 Oct 23 05:07:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 05:12:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 39232eca-8a6f-d1da-2c31-841feb0155ce (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf36d7000, cur 1540296771 expire 1540296621 last 1540296544 Oct 23 05:14:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 05:18:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 05:19:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a63ec74e-e7ed-031a-d360-b7a0eb8175c3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea84d9c00, cur 1540297157 expire 1540297007 last 1540296930 Oct 23 05:24:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 05:24:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 623f9f71-6671-a3a7-bd78-f479ee955b2c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88180feb1000, cur 1540297450 expire 1540297300 last 1540297223 Oct 23 05:31:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 06e980cf-e9e7-4a8e-7320-43d3e25f5c9a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25733400, cur 1540297872 expire 1540297722 last 1540297645 Oct 23 05:35:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2ce0e707-77ed-65c6-f694-3970a22183f1 (at 10.8.1.24@o2ib6) reconnecting Oct 23 05:35:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ed11cf00-fceb-e546-a223-07cb44ae6867 (at 10.8.1.24@o2ib6) Oct 23 05:35:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 05:43:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client abc033ea-4a3f-3d5c-8dfe-72bda5a0cd07 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a16a80400, cur 1540298600 expire 1540298450 last 1540298373 Oct 23 05:43:20 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 05:49:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 05:49:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 05:54:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5054c155-b189-0140-f214-0f256938c7a4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e75232c00, cur 1540299276 expire 1540299126 last 1540299049 Oct 23 05:54:36 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 06:06:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 06:06:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 06:09:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 0cc30097-9c10-00eb-665b-baad25ade550 (at 10.8.18.3@o2ib6) reconnecting Oct 23 06:10:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0dc6639-2b7c-a0d9-f7c5-fb1f2234a863 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880050f9dc00, cur 1540300219 expire 1540300069 last 1540299992 Oct 23 06:10:19 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 06:19:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 06:19:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 06:23:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0cb0c8b8-ab44-1443-06d3-c802aa64e327 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000370a800, cur 1540301008 expire 1540300858 last 1540300781 Oct 23 06:23:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 06:30:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 06:30:17 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 06:34:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b4b661da-b7b1-4198-1922-5b20f6c30829 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039f59f000, cur 1540301644 expire 1540301494 last 1540301417 Oct 23 06:34:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 06:42:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 06:42:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 06:45:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a73eb654-fe1d-2fe5-6d4d-a1684cb61c53 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851235c00, cur 1540302347 expire 1540302197 last 1540302120 Oct 23 06:45:47 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 06:55:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9888d9d3-bd22-50ef-852a-267dbcfc00a4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba6400, cur 1540302958 expire 1540302808 last 1540302731 Oct 23 06:55:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 07:00:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 07:00:51 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 07:17:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 07:17:23 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 07:18:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7a5c8e7-87d2-b493-6e08-948e0e8cf074 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88187a309000, cur 1540304281 expire 1540304131 last 1540304054 Oct 23 07:18:01 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 07:24:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93e53165-196c-566b-3e42-13abcc941772 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880860fa1400, cur 1540304645 expire 1540304495 last 1540304418 Oct 23 07:28:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bcff3b16-7f04-d593-bc0c-54eba330a657 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0fac00, cur 1540304889 expire 1540304739 last 1540304662 Oct 23 07:28:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 07:28:12 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 07:36:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce7e9cdd-ca9f-1b20-db21-8b277af96bc1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881558a5f800, cur 1540305397 expire 1540305247 last 1540305170 Oct 23 07:36:37 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 07:41:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 07:41:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 07:47:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5a15f605-f029-c8ef-788a-c3bd2c94fa7c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801530ecc00, cur 1540306079 expire 1540305929 last 1540305852 Oct 23 07:47:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 07:52:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 3d04dae4-9c72-8809-bcff-e9d10e0ad154 (at 10.8.15.5@o2ib6) Oct 23 07:52:39 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 23 07:58:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client faaf657a-1f1d-3536-61c0-4b2056adbbfc (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881030654400, cur 1540306682 expire 1540306532 last 1540306455 Oct 23 07:58:02 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 23 08:02:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 08:02:50 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 08:12:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b39f72ae-2866-608b-46fa-4dd02414d150 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a98403c00, cur 1540307561 expire 1540307411 last 1540307334 Oct 23 08:12:41 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 23 08:14:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 08:14:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 08:29:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49b7a393-dd3c-702d-1cb3-1b193a583cb6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880530fdec00, cur 1540308551 expire 1540308401 last 1540308324 Oct 23 08:29:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 08:31:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 08:46:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 08:46:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 08:47:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 75ef7613-fc65-fc6c-1ebf-984cda74af2c (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880272371000, cur 1540309678 expire 1540309528 last 1540309451 Oct 23 08:47:58 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 09:10:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 09:10:33 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 23 09:10:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 52e170e3-66cf-931e-17aa-f38271082749 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880050f99c00, cur 1540311048 expire 1540310898 last 1540310821 Oct 23 09:10:48 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 09:30:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 09:31:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c3ddcd5e-f2eb-fc9f-857d-14df8de560a1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a8ca99800, cur 1540312285 expire 1540312135 last 1540312058 Oct 23 09:54:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0884779a-f472-2adf-ff8c-5546a1b2552a (at 10.8.2.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ee7550c00, cur 1540313685 expire 1540313535 last 1540313458 Oct 23 10:09:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 10:10:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 777b4104-70f5-5518-c5a3-d30f6ecf259a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dacb96000, cur 1540314640 expire 1540314490 last 1540314413 Oct 23 10:20:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8a737cd1-8357-eca0-2471-04359d20cdac (at 10.8.2.18@o2ib6) Oct 23 10:33:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8763b52d-0edf-b0e1-1b9d-ddb752f9bb61 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880965ab0000, cur 1540315994 expire 1540315844 last 1540315767 Oct 23 10:34:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 10:45:25 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540316717/real 1540316717] req@ffff880d35d91500 x1610635316819504/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540316724 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 23 10:45:25 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Oct 23 10:45:32 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540316725/real 1540316725] req@ffff880d35d91500 x1610635316819504/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540316732 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 23 10:45:39 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540316732/real 1540316732] req@ffff880d35d91500 x1610635316819504/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540316739 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 23 10:45:53 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540316746/real 1540316746] req@ffff880d35d91500 x1610635316819504/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540316753 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 23 10:45:53 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 23 10:46:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 10:46:14 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540316767/real 1540316767] req@ffff880d35d91500 x1610635316819504/t0(0) o104->oak-MDT0000@10.8.18.35@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540316774 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 23 10:46:14 oak-md1-s2 kernel: Lustre: 3867:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 23 10:46:14 oak-md1-s2 kernel: LustreError: 3867:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.18.35@o2ib6) returned error from blocking AST (req@ffff880d35d91500 x1610635316819504 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8814b4613a00/0x6c00c4479b76988f lrc: 4/0,0 mode: PR/PR res: [0x200013483:0x19b76:0x0].0x0 bits 0x13 rrc: 15 type: IBT flags: 0x60200400000020 nid: 10.8.18.35@o2ib6 remote: 0x666733e2db707818 expref: 13 pid: 10665 timeout: 8601117499 lvb_type: 0 Oct 23 10:46:14 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.8.18.35@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Oct 23 11:10:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 18a04912-7ed3-aba1-9728-f58b2de2eb6d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daa972c00, cur 1540318209 expire 1540318059 last 1540317982 Oct 23 11:12:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 11:34:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7c6aaf34-f51e-0d40-28b6-941fb03490d3 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880de03a2000, cur 1540319644 expire 1540319494 last 1540319417 Oct 23 11:38:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4b4666da-ba99-1226-e4c3-6c436ffc8157 (at 10.12.4.22@o2ib) Oct 23 11:45:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b56afd7e-48af-b641-e2a6-e7ae4327c82a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e75231800, cur 1540320348 expire 1540320198 last 1540320121 Oct 23 11:46:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 11:48:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 799a7f96-3da7-e2de-abe5-4ecf813f725a (at 10.12.4.22@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8801b7b17400, cur 1540320526 expire 1540320376 last 1540320299 Oct 23 11:48:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 23 11:50:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f908562c-efaf-a94a-04ad-57aa7160a39c (at 10.8.20.15@o2ib6) in 223 seconds. I think it's dead, and I am evicting it. exp ffff880066e5a800, cur 1540320602 expire 1540320452 last 1540320379 Oct 23 12:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 12:18:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dddb72d5-c68a-06ff-3c68-372d707b925f (at 10.8.2.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68e000, cur 1540322323 expire 1540322173 last 1540322096 Oct 23 12:43:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ee971b0c-000c-ebac-15f1-20b1fab3c8ff (at 10.8.2.5@o2ib6) Oct 23 12:44:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9be6edfa-ebde-1d5d-f312-2249d8fda40b (at 10.8.2.11@o2ib6) Oct 23 13:09:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0a8ada4-a91d-a8f5-e7df-62a9d18cc013 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005e12b000, cur 1540325362 expire 1540325212 last 1540325135 Oct 23 13:09:22 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 13:20:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 13:21:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 331319fd-9446-8eb8-85ea-86ba29a1fee0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804fb5c7000, cur 1540326065 expire 1540325915 last 1540325838 Oct 23 13:25:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a923e00f-1e6b-77c1-bcec-7ebcec0fcc31 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dab42a000, cur 1540326343 expire 1540326193 last 1540326116 Oct 23 13:26:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 13:30:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c225ee44-2607-5051-8039-ccd696d5c1ff (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019e610c00, cur 1540326622 expire 1540326472 last 1540326395 Oct 23 13:30:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 13:35:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 72b47082-f20a-9620-9ffa-ba5939082b37 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881bf1ea0800, cur 1540326905 expire 1540326755 last 1540326678 Oct 23 13:35:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 13:37:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7592846a-577d-ad5f-ca2f-3277b2c7f998 (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e95c98400, cur 1540327072 expire 1540326922 last 1540326845 Oct 23 13:39:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9019f48-08d3-ae06-99eb-9b95ea2bcfe3 (at 10.8.18.35@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff88022a131000, cur 1540327148 expire 1540326998 last 1540326977 Oct 23 13:40:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 13:46:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79479b75-faa4-aac9-699a-0b5ae4cbe32d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881cdc706000, cur 1540327574 expire 1540327424 last 1540327347 Oct 23 13:47:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 13:52:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800178fcc00, cur 1540327969 expire 1540327819 last 1540327742 Oct 23 13:54:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 60d2a6f3-c520-6f72-8b1b-2840bfe4db3f (at 10.8.18.35@o2ib6) in 216 seconds. I think it's dead, and I am evicting it. exp ffff880ee1a00800, cur 1540328045 expire 1540327895 last 1540327829 Oct 23 13:54:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 13:54:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 13:58:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ccf2dd9e-c839-162a-2b8e-1c523e2477f2 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880779d39c00, cur 1540328316 expire 1540328166 last 1540328089 Oct 23 13:59:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 14:05:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f75fe318-5fed-1a57-f17d-5cc717974069 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819f3aabc00, cur 1540328715 expire 1540328565 last 1540328488 Oct 23 14:05:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 14:05:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 14:11:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 14:11:00 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 14:16:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 386c52d0-fc0c-a1b5-0307-094055c48f0a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881868a08800, cur 1540329412 expire 1540329262 last 1540329185 Oct 23 14:16:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 14:24:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 14:24:08 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 14:27:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ce86adb6-66af-5ae0-b3b7-0abb8cb766e3 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d169ce400, cur 1540330062 expire 1540329912 last 1540329835 Oct 23 14:27:42 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 23 14:35:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 23 14:35:33 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 14:41:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a896e332-d50b-9f8f-08ee-3f521f5d4074 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881520e6b800, cur 1540330860 expire 1540330710 last 1540330633 Oct 23 14:41:00 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 23 14:59:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 14:59:10 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 15:12:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client db23d29e-40cc-8978-48df-e6486a1b4501 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88022a136c00, cur 1540332776 expire 1540332626 last 1540332549 Oct 23 15:18:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 15:19:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a2d543b0-fd57-608a-54fb-230f56b88cf1 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e5f68bc00, cur 1540333191 expire 1540333041 last 1540332964 Oct 23 15:36:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27bac3ac-d257-a563-4125-c911a485eb0b (at 10.9.101.60@o2ib4) Oct 23 15:38:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27bac3ac-d257-a563-4125-c911a485eb0b (at 10.9.101.60@o2ib4) Oct 23 15:43:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7f39559c-45f9-2121-522d-94f053cf24fa (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800567bec00, cur 1540334608 expire 1540334458 last 1540334381 Oct 23 15:43:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 15:56:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59744830-ad33-2648-2c0f-8a4361849402 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b11862000, cur 1540335362 expire 1540335212 last 1540335135 Oct 23 15:56:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 16:04:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0472e2c8-cc65-b4a4-1bed-f5b3ce8d88f6 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880003ac0000, cur 1540335850 expire 1540335700 last 1540335623 Oct 23 16:14:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 16:21:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client cf1ee784-b117-d4b8-80b8-f52fff0ae206 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f93136800, cur 1540336911 expire 1540336761 last 1540336684 Oct 23 16:30:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 16:35:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 16:36:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2a0f2836-7ca9-2dfa-a725-0443fcc8a209 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2114c00, cur 1540337760 expire 1540337610 last 1540337533 Oct 23 16:42:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 16:42:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ee5c66c-815c-cc03-970b-178525be7adf (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802b68e0400, cur 1540338178 expire 1540338028 last 1540337951 Oct 23 16:48:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 45fa25c0-41b9-2745-8909-d77ee8d18c4e (at 10.9.107.40@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804eee27400, cur 1540338536 expire 1540338386 last 1540338309 Oct 23 16:49:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 16:53:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 16:53:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a30e1d93-0527-867a-be7b-4a5b4ce55209 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805adfaa800, cur 1540338812 expire 1540338662 last 1540338585 Oct 23 16:53:32 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 17:00:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 96940744-2471-7e4d-c16d-10786b53904e (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88029d918c00, cur 1540339246 expire 1540339096 last 1540339019 Oct 23 17:01:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 17:04:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 45fa25c0-41b9-2745-8909-d77ee8d18c4e (at 10.9.107.40@o2ib4) Oct 23 17:08:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ff4aea3d-580b-fb1c-1138-f753ddb516bd (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fd8afc000, cur 1540339713 expire 1540339563 last 1540339486 Oct 23 17:09:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 17:12:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a424acbc-5263-a91a-56e7-faa09e54a6e4 (at 10.9.107.47@o2ib4) Oct 23 17:18:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) Oct 23 17:18:01 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 23 17:24:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ef8782000, cur 1540340658 expire 1540340508 last 1540340431 Oct 23 17:25:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 17:50:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4021d1cd-e131-c8e2-e47d-2b937d8dc85c (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880041c1b000, cur 1540342213 expire 1540342063 last 1540341986 Oct 23 17:50:13 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 23 17:51:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 17:57:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0bad5f33-e37c-c099-4cf2-d4df3131c732 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88059c299000, cur 1540342666 expire 1540342516 last 1540342439 Oct 23 17:58:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 23 18:02:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b71b0ef2-d77b-ba3b-98b4-fc7330d2a25b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beea0fc00, cur 1540342953 expire 1540342803 last 1540342726 Oct 23 23:24:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 27c1e069-126a-681e-a8e5-f554b5159156 (at 10.8.26.35@o2ib6) reconnecting Oct 23 23:24:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aee4869e-c229-6e37-2dde-13e266d0fbc7 (at 10.8.26.35@o2ib6) Oct 24 00:47:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client daf17879-02bf-d70a-73f4-cb40e30e3ebc (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880053efc800, cur 1540367240 expire 1540367090 last 1540367013 Oct 24 01:26:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 4f8ff9b5-3c9f-09af-6ef9-a813290ae6d7 (at 10.8.14.8@o2ib6) Oct 24 03:02:28 oak-md1-s2 kernel: LNet: 322993:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.245@o2ib5 version 12/12 incarnation 1539372370411081/1540375346347635 Oct 24 03:02:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fccddc08-d568-f4b4-6a44-f4e21b04be63 (at 10.0.2.245@o2ib5) Oct 24 03:05:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b0619131-12ff-e061-3142-0d40f3c044f6 (at 10.0.2.245@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200c5a0000, cur 1540375510 expire 1540375360 last 1540375283 Oct 24 07:55:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) Oct 24 08:14:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 9b698a16-09ef-aa06-c3c2-58fccf88791e (at 10.9.0.2@o2ib4) Oct 24 08:21:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48a5fbdf-671a-df41-cea1-dc4ee1d9f555 (at 10.8.15.4@o2ib6) Oct 24 08:23:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02e61f71-becc-75cc-c72e-8197c3ec95b8 (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe7594c00, cur 1540394639 expire 1540394489 last 1540394412 Oct 24 08:53:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d8089568-69e3-10f9-beca-8489e6201ad3 (at 10.9.112.13@o2ib4) Oct 24 08:53:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 08:56:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 24 08:57:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 24 10:24:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0730a9f5-4077-2336-0ed5-d2de5f802c0a (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880003ac7800, cur 1540401877 expire 1540401727 last 1540401650 Oct 24 10:24:37 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 24 10:44:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27bac3ac-d257-a563-4125-c911a485eb0b (at 10.9.101.60@o2ib4) Oct 24 10:58:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 11:06:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 29e26516-905b-3654-2e71-6d502176cd62 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880819d32400, cur 1540404380 expire 1540404230 last 1540404153 Oct 24 11:15:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 24 11:15:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e56c590a-5442-dd9a-8161-244f65ac07b5 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815972e4800, cur 1540404949 expire 1540404799 last 1540404722 Oct 24 11:18:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 11:22:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b3e8fc83-d360-1f41-0f42-23524a705364 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805f1815000, cur 1540405326 expire 1540405176 last 1540405099 Oct 24 11:24:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 11:27:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0d0a4dda-d6b1-2a80-b8d4-71c4511758a3 (at 10.9.105.66@o2ib4) Oct 24 11:28:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c7cf36f1-a659-4ad0-3bc6-f25fb17db4b8 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802e3c24400, cur 1540405684 expire 1540405534 last 1540405457 Oct 24 11:42:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 24 11:42:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 300a37c0-bb5d-e265-a1de-90a94a4a8120 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001f5a0000, cur 1540406558 expire 1540406408 last 1540406331 Oct 24 11:42:38 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 11:50:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 24 11:50:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9583b17c-4e15-307b-27f6-1a6a598d4daa (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815bc231800, cur 1540407022 expire 1540406872 last 1540406795 Oct 24 11:52:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 24 11:53:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 92e4aefd-1a04-9a27-82fa-26fe1a293671 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8818692dd800, cur 1540407208 expire 1540407058 last 1540406981 Oct 24 12:02:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 12:16:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4ecb14d3-c6e7-c1a8-693b-1e5d28b248ce (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881beeebe000, cur 1540408563 expire 1540408413 last 1540408336 Oct 24 12:53:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d03b40b9-24dd-18d7-2a84-8e187ce48104 (at 10.12.4.48@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880046e2a000, cur 1540410791 expire 1540410641 last 1540410564 Oct 24 13:08:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 13:10:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 24 13:17:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8ebd7df0-2c48-0eda-2f5a-5bdf7d2df2ca (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880a1f6ae800, cur 1540412276 expire 1540412126 last 1540412049 Oct 24 13:25:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 13:35:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d519dcc1-7328-bc86-5fb9-90174741d6ec (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88019e610c00, cur 1540413337 expire 1540413187 last 1540413110 Oct 24 13:36:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8443d524-8d55-009e-1371-5359ce39ea80 (at 10.8.18.35@o2ib6) in 219 seconds. I think it's dead, and I am evicting it. exp ffff8812cbf50400, cur 1540413413 expire 1540413263 last 1540413194 Oct 24 13:41:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 13:45:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fb9477e6-9d3e-bd16-ae1b-72ba8d1320dd (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea36fa800, cur 1540413916 expire 1540413766 last 1540413689 Oct 24 13:45:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 694f22d4-b8d2-14f7-86aa-c36888a919bf (at 10.8.1.29@o2ib6) Oct 24 13:46:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 24 13:50:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7a712aa2-794b-77e2-52af-8b45b92cc98a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b0122b400, cur 1540414235 expire 1540414085 last 1540414008 Oct 24 13:54:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 13:55:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 047b5a85-a11d-e9d5-c739-1a4c2c0b792e (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88000370d800, cur 1540414505 expire 1540414355 last 1540414278 Oct 24 14:01:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5ce6f721-2d4b-6284-4fe7-f5349d228cf8 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805adfae800, cur 1540414873 expire 1540414723 last 1540414646 Oct 24 14:01:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 14:11:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f1ff3100-5869-4e7c-1af8-8b682f6ed4fe (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88039f599800, cur 1540415483 expire 1540415333 last 1540415256 Oct 24 14:12:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 14:22:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 14:22:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 050065f1-69d0-0059-e94f-13a0fbcc35d1 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0fe400, cur 1540416174 expire 1540416024 last 1540415947 Oct 24 14:27:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) Oct 24 14:33:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98565b50-3fec-e171-4093-62cf7c1daa26 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfefc7800, cur 1540416828 expire 1540416678 last 1540416601 Oct 24 14:34:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 14:39:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 67533ae6-18d9-00cf-74a0-166d6fc6ab0a (at 10.12.5.39@o2ib) Oct 24 14:52:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 67533ae6-18d9-00cf-74a0-166d6fc6ab0a (at 10.12.5.39@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b5f37c800, cur 1540417921 expire 1540417771 last 1540417694 Oct 24 14:52:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 15:10:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 99939326-c866-0495-da5e-e0f491029360 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88096a0f4400, cur 1540419028 expire 1540418878 last 1540418801 Oct 24 15:10:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 24 15:32:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 15:58:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) Oct 24 16:06:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 24 16:07:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 58497426-bd5d-0b81-8a25-e4c757b762f2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d10a53c00, cur 1540422421 expire 1540422271 last 1540422194 Oct 24 16:16:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c38b46b7-6fbb-3eaf-4c33-849a1d96a31b (at 10.12.5.38@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880013a61000, cur 1540423019 expire 1540422869 last 1540422792 Oct 24 16:21:31 oak-md1-s2 kernel: Lustre: 3805:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540423075/real 1540423075] req@ffff881b6addc200 x1610635660481296/t0(0) o6->oak-OST0009-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 664/432 e 0 to 1 dl 1540423291 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 24 16:21:31 oak-md1-s2 kernel: Lustre: oak-OST0009-osc-MDT0000: Connection to oak-OST0009 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 24 16:21:31 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 24 16:21:38 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1540423291/real 0] req@ffff881cd2a13000 x1610635660626336/t0(0) o8->oak-OST0009-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1540423298 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 24 16:22:05 oak-md1-s2 kernel: Lustre: 3803:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540423078/real 1540423078] req@ffff8811e8f6ce00 x1610635660484448/t0(0) o6->oak-OST0009-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 664/432 e 0 to 1 dl 1540423325 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 24 16:22:07 oak-md1-s2 kernel: Lustre: oak-OST002f-osc-MDT0000: Connection to oak-OST002f (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 24 16:22:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 9ea52517-e6c8-5215-6de6-a80ffe09035a (at 10.9.101.30@o2ib4) reconnecting Oct 24 16:22:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 291d7bea-f250-4d15-130f-a2b94cc43882 (at 10.9.101.30@o2ib4) Oct 24 16:22:09 oak-md1-s2 kernel: Lustre: oak-OST0003-osc-MDT0000: Connection to oak-OST0003 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 24 16:22:09 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 24 16:22:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 2655fc21-5399-427c-a760-b0b9111b4a19 (at 10.9.105.7@o2ib4) reconnecting Oct 24 16:22:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 16:22:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 44b1a1e5-37ba-7a23-b409-18d01f89c7f8 (at 10.9.105.7@o2ib4) Oct 24 16:22:09 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 16:22:11 oak-md1-s2 kernel: Lustre: oak-OST001b-osc-MDT0000: Connection to oak-OST001b (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 24 16:22:11 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Oct 24 16:22:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8b475bfa-deaf-641f-f6e8-d1a416159ba0 (at 10.8.18.13@o2ib6) reconnecting Oct 24 16:22:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 16:22:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f28d8bfc-6c86-b70e-0a23-c5732fa66f3d (at 10.8.18.13@o2ib6) Oct 24 16:22:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 16:22:15 oak-md1-s2 kernel: Lustre: oak-OST002b-osc-MDT0000: Connection to oak-OST002b (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete Oct 24 16:22:15 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 24 16:22:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client cd7a2eac-6d83-8ad7-5e17-cbeafe5b3a22 (at 10.8.12.15@o2ib6) reconnecting Oct 24 16:22:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 24 16:22:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 6af7bd86-edb2-57a2-48a8-0cebacc5994a (at 10.8.12.15@o2ib6) Oct 24 16:22:16 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 24 16:22:18 oak-md1-s2 kernel: LNetError: 3778:0:(o2iblnd_cb.c:3147:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 2 seconds Oct 24 16:22:18 oak-md1-s2 kernel: LNetError: 3778:0:(o2iblnd_cb.c:3222:kiblnd_check_conns()) Timed out RDMA with 10.0.2.102@o2ib5 (48): c: 0, oc: 3, rc: 8 Oct 24 16:22:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client d2eecc02-85e5-9a1f-be57-511e7db8d8d6 (at 10.8.7.3@o2ib6) reconnecting Oct 24 16:22:20 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 24 16:22:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0e6f9fc9-687c-39ae-a0de-c2df30c5a9d7 (at 10.8.7.3@o2ib6) Oct 24 16:22:20 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 24 16:22:21 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1540423335/real 0] req@ffff881f6b1faa00 x1610635660638224/t0(0) o8->oak-OST002b-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1540423341 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 24 16:22:21 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Oct 24 16:22:30 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 13 seconds Oct 24 16:22:30 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 4 previous similar messages Oct 24 16:22:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 3d7826f8-1bdb-8d0c-24bf-ae56d1d9f355 (at 10.8.13.15@o2ib6) reconnecting Oct 24 16:22:30 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 24 16:22:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a2ae4c69-d8f4-5e07-dd5a-faad2385cbc1 (at 10.8.13.15@o2ib6) Oct 24 16:22:30 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 24 16:22:37 oak-md1-s2 kernel: LustreError: 10854:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.8.6.36@o2ib6 arrived at 1540423357 with bad export cookie 0 Oct 24 16:22:43 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 26 seconds Oct 24 16:22:43 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 4 previous similar messages Oct 24 16:22:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 022edf6b-221d-9b06-81e4-85c768eae902 (at 10.9.101.59@o2ib4) reconnecting Oct 24 16:22:52 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 24 16:22:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Oct 24 16:22:52 oak-md1-s2 kernel: Lustre: Skipped 6 previous similar messages Oct 24 16:23:08 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 51 seconds Oct 24 16:23:08 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 7 previous similar messages Oct 24 16:23:08 oak-md1-s2 kernel: Lustre: 3813:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1540423316/real 1540423388] req@ffff881cd2a12d00 x1610635660632784/t0(0) o400->oak-OST000d-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 224/224 e 0 to 1 dl 1540423719 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 24 16:23:08 oak-md1-s2 kernel: Lustre: 3813:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 24 16:23:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 646cefe6-a858-5c13-7bf0-67db87dd4925 (at 10.9.105.66@o2ib4) reconnecting Oct 24 16:23:29 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 24 16:23:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 0d0a4dda-d6b1-2a80-b8d4-71c4511758a3 (at 10.9.105.66@o2ib4) Oct 24 16:23:29 oak-md1-s2 kernel: Lustre: Skipped 11 previous similar messages Oct 24 16:23:46 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 37 seconds Oct 24 16:23:46 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 12 previous similar messages Oct 24 16:24:56 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1540423485/real 0] req@ffff881599b5bc00 x1610635660678656/t0(0) o8->oak-OST002f-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1540423496 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 24 16:24:56 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Oct 24 16:25:02 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 1 seconds Oct 24 16:25:02 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 26 previous similar messages Oct 24 16:25:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 16:25:24 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Oct 24 16:27:06 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1540423610/real 0] req@ffff881599b58900 x1610635660711376/t0(0) o8->oak-OST002f-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1540423626 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 24 16:27:06 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Oct 24 16:27:33 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 65 seconds Oct 24 16:27:33 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 32 previous similar messages Oct 24 16:31:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) Oct 24 16:31:31 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1540423860/real 0] req@ffff881a9a854b00 x1610635660777232/t0(0) o8->oak-OST002f-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1540423891 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Oct 24 16:31:31 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Oct 24 16:33:14 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 8 seconds Oct 24 16:33:14 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 55 previous similar messages Oct 24 16:40:11 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1540424360/real 1540424411] req@ffff881b92ab0900 x1610635660909104/t0(0) o8->oak-OST002d-osc-MDT0000@10.0.2.102@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1540424415 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Oct 24 16:40:11 oak-md1-s2 kernel: Lustre: 3790:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 140 previous similar messages Oct 24 16:43:20 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 10.0.2.102@o2ib5: 20 seconds Oct 24 16:43:20 oak-md1-s2 kernel: LNet: 3778:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 142 previous similar messages Oct 24 16:44:38 oak-md1-s2 kernel: LustreError: 167-0: oak-OST000b-osc-MDT0000: This client was evicted by oak-OST000b; in progress operations using this service will fail. Oct 24 16:44:38 oak-md1-s2 kernel: Lustre: oak-OST000b-osc-MDT0000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5) Oct 24 16:44:38 oak-md1-s2 kernel: Lustre: Skipped 22 previous similar messages Oct 24 16:44:41 oak-md1-s2 kernel: LustreError: 167-0: oak-OST0025-osc-MDT0000: This client was evicted by oak-OST0025; in progress operations using this service will fail. Oct 24 16:44:41 oak-md1-s2 kernel: LustreError: Skipped 21 previous similar messages Oct 24 16:44:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.102@o2ib5, removing former export from same NID Oct 24 16:44:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.102@o2ib5, removing former export from same NID Oct 24 16:44:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.102@o2ib5, removing former export from same NID Oct 24 16:44:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.102@o2ib5, removing former export from same NID Oct 24 16:44:43 oak-md1-s2 kernel: LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.0.2.102@o2ib5 (no target). If you are running an HA pair check that the target is mounted on the other server. Oct 24 16:44:43 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 24 16:44:45 oak-md1-s2 kernel: LustreError: 167-0: oak-OST0007-osc-MDT0000: This client was evicted by oak-OST0007; in progress operations using this service will fail. Oct 24 16:45:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Received new LWP connection from 10.0.2.102@o2ib5, removing former export from same NID Oct 24 16:45:08 oak-md1-s2 kernel: LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.0.2.102@o2ib5 (no target). If you are running an HA pair check that the target is mounted on the other server. Oct 24 16:45:08 oak-md1-s2 kernel: LustreError: Skipped 6 previous similar messages Oct 24 16:45:08 oak-md1-s2 kernel: Lustre: Skipped 15 previous similar messages Oct 24 16:48:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 51c2307b-10da-ffd3-94bc-9c2e366be442 (at 10.12.5.38@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b5f37c800, cur 1540424930 expire 1540424780 last 1540424703 Oct 24 16:48:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 16:52:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 83394eb5-e321-8e43-3d0e-4fceccf795af (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b5f37c000, cur 1540425176 expire 1540425026 last 1540424949 Oct 24 16:52:56 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 16:55:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 16:55:10 oak-md1-s2 kernel: Lustre: Skipped 26 previous similar messages Oct 24 17:07:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e40ede5d-8176-a053-dd7e-51c43eae12fc (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8807e2111800, cur 1540426063 expire 1540425913 last 1540425836 Oct 24 17:13:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 17:20:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 642ed576-5771-4792-eb2d-a9c08187caeb (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881037d47800, cur 1540426844 expire 1540426694 last 1540426617 Oct 24 17:25:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2b9cc1e0-3b38-25f5-8a65-91f860a49647 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d1a248400, cur 1540427157 expire 1540427007 last 1540426930 Oct 24 17:27:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 17:27:12 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 17:39:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0d599571-5d00-da19-f608-c208c154e803 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b95222800, cur 1540427949 expire 1540427799 last 1540427722 Oct 24 17:39:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 24 17:44:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7567e89-aac6-e63b-172d-a143f9b369b4 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880fb0964400, cur 1540428284 expire 1540428134 last 1540428057 Oct 24 17:50:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 17:50:26 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 17:59:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0029d049-bb0a-946f-4f79-8086787253b2 (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881af267e000, cur 1540429147 expire 1540428997 last 1540428920 Oct 24 18:21:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d1c1ea3-4c1a-1b37-b15a-86bd27a46519 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803c3afa400, cur 1540430503 expire 1540430353 last 1540430276 Oct 24 19:05:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 24 19:05:07 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 19:05:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6dda3c43-dce0-90d9-d0d8-a5a13ba909a4 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810166a4c00, cur 1540433109 expire 1540432959 last 1540432882 Oct 24 19:32:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 19:48:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5bd9ada1-1710-398a-d169-f7596afdbd3c (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bfefc0c00, cur 1540435715 expire 1540435565 last 1540435488 Oct 24 19:49:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 19:53:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c2aeae68-aa6f-809f-99b4-5a392497f34a (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805d4fcc000, cur 1540436011 expire 1540435861 last 1540435784 Oct 24 20:14:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 20:46:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5315a82a-afa0-867c-a5e3-84c9661cffa2 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bbaff800, cur 1540439216 expire 1540439066 last 1540438989 Oct 24 20:48:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 20:58:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f8c5ca0-2564-36da-4d3f-4c70e1a71f0d (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227ce800, cur 1540439908 expire 1540439758 last 1540439681 Oct 24 21:18:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d0cf9c14-f599-1396-8e2b-7f426fe1bf4a (at 10.8.15.8@o2ib6) Oct 24 21:19:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3e5f5511-5849-4252-6aac-f6212a574a44 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880050f98400, cur 1540441185 expire 1540441035 last 1540440958 Oct 24 21:51:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 22:00:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) Oct 24 22:08:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 67533ae6-18d9-00cf-74a0-166d6fc6ab0a (at 10.12.5.39@o2ib) Oct 24 22:08:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34a25f80-7263-29af-b47a-e1d4c4b2d08a (at 10.12.5.40@o2ib) Oct 24 22:09:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8e6255b2-c53d-43fa-e3b1-899cb60f40be (at 10.12.5.41@o2ib) Oct 24 22:09:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 935b9637-e220-145f-2f69-66fa9dcbe8c2 (at 10.12.4.30@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880052c86000, cur 1540444162 expire 1540444012 last 1540443935 Oct 24 22:10:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0757d9fb-f472-0099-fd12-8f8b20eac7c4 (at 10.12.4.43@o2ib) in 154 seconds. I think it's dead, and I am evicting it. exp ffff880530fda800, cur 1540444238 expire 1540444088 last 1540444084 Oct 24 22:11:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f9ead67e-9fa0-bff3-a547-d355b730ac60 (at 10.12.4.41@o2ib) in 224 seconds. I think it's dead, and I am evicting it. exp ffff8800561be800, cur 1540444314 expire 1540444164 last 1540444090 Oct 24 22:12:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 96cb526e-b16d-4292-ee35-19453749e5b6 (at 10.12.5.42@o2ib) Oct 24 22:13:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e786f2d2-6920-6133-2949-f36784038214 (at 10.12.4.45@o2ib) in 221 seconds. I think it's dead, and I am evicting it. exp ffff880036d67000, cur 1540444390 expire 1540444240 last 1540444169 Oct 24 22:36:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 20696ab3-77f6-e423-0a18-5be4dca7c662 (at 10.12.5.44@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803352b5400, cur 1540445805 expire 1540445655 last 1540445578 Oct 24 22:41:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 20696ab3-77f6-e423-0a18-5be4dca7c662 (at 10.12.5.44@o2ib) Oct 24 22:41:28 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 24 22:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client b63921a6-5286-2ae9-b71d-9ae6240c77b4 (at 10.8.26.33@o2ib6) reconnecting Oct 24 22:43:09 oak-md1-s2 kernel: Lustre: Skipped 16 previous similar messages Oct 24 22:43:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bcf31f1d-770a-6e80-923f-b21b7ce98271 (at 10.8.26.33@o2ib6) Oct 24 23:01:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 20696ab3-77f6-e423-0a18-5be4dca7c662 (at 10.12.5.44@o2ib) Oct 24 23:01:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f7a3be54-f767-0835-9787-85c1adca83ed (at 10.12.5.44@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880530fda000, cur 1540447291 expire 1540447141 last 1540447064 Oct 24 23:12:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 20696ab3-77f6-e423-0a18-5be4dca7c662 (at 10.12.5.44@o2ib) Oct 24 23:12:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 887971ba-f2ff-d249-16fa-1b7cc9e32dbf (at 10.12.5.44@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88068aa28800, cur 1540447946 expire 1540447796 last 1540447719 Oct 24 23:22:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a97ac5b8-979e-0386-d35c-c33acc053436 (at 10.12.5.43@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f56b16800, cur 1540448579 expire 1540448429 last 1540448352 Oct 24 23:24:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8e6255b2-c53d-43fa-e3b1-899cb60f40be (at 10.12.5.41@o2ib) in 219 seconds. I think it's dead, and I am evicting it. exp ffff88055c148000, cur 1540448655 expire 1540448505 last 1540448436 Oct 24 23:24:15 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 24 23:26:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 23:30:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4a259f8-fd4f-3c1c-790e-27b1024cea98 (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880aa6e44c00, cur 1540449037 expire 1540448887 last 1540448810 Oct 24 23:30:37 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 24 23:31:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 23:35:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d38f9609-1e28-5056-6705-1df96e7f2dbc (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880392e3a800, cur 1540449322 expire 1540449172 last 1540449095 Oct 24 23:37:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 23:41:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a79290e9-b2e1-5566-1f5c-4e276e9d6a6f (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b29bfc800, cur 1540449676 expire 1540449526 last 1540449449 Oct 24 23:41:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 23:45:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 60d940b7-64a5-a59a-69bf-ae26cc40e18b (at 10.12.5.37@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d28641800, cur 1540449924 expire 1540449774 last 1540449697 Oct 24 23:46:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 24 23:50:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) Oct 24 23:54:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34a25f80-7263-29af-b47a-e1d4c4b2d08a (at 10.12.5.40@o2ib) Oct 25 00:24:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 00:24:01 oak-md1-s2 kernel: Lustre: Skipped 5 previous similar messages Oct 25 00:25:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client acc12881-8456-d43a-45d1-9e0df36c6e37 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dac8fc800, cur 1540452300 expire 1540452150 last 1540452073 Oct 25 00:28:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2d450def-5ac6-f307-e6f3-a1c23e5ce793 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039230000, cur 1540452500 expire 1540452350 last 1540452273 Oct 25 00:29:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 68e6dd80-cf93-c8b6-a82a-90ebf544c508 (at 10.8.28.6@o2ib6) in 177 seconds. I think it's dead, and I am evicting it. exp ffff8814b268f000, cur 1540452576 expire 1540452426 last 1540452399 Oct 25 00:30:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 31b281e2-0364-cda8-4967-e86b1a625181 (at 10.8.28.6@o2ib6) Oct 25 00:35:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 01:02:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2851dda1-b02b-9562-31b6-9e003c2e2415 (at 10.8.18.35@o2ib6) Oct 25 01:06:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 65525c39-ef89-f3e8-b926-d3204474aee3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880040ed0000, cur 1540454787 expire 1540454637 last 1540454560 Oct 25 01:14:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 01:15:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e28b34e-99ba-d3c9-e292-8da61d302d3f (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880038e8b400, cur 1540455307 expire 1540455157 last 1540455080 Oct 25 01:18:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 01:19:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2e58da33-d32b-f1fb-c3e1-65b32fda0a0b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88189aa8cc00, cur 1540455558 expire 1540455408 last 1540455331 Oct 25 01:22:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 01:22:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client cff326e8-be94-090a-03cc-3f71d1d3a553 (at 10.8.17.14@o2ib6) reconnecting Oct 25 01:22:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 058b6278-177a-f759-6963-9e13604935c3 (at 10.8.17.14@o2ib6) Oct 25 01:23:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f4851a44-971d-8fab-ac26-a2d5be9cc322 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e88ee0400, cur 1540455783 expire 1540455633 last 1540455556 Oct 25 01:38:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b08afd88-5b5a-4e15-bdef-24d1df016dbe (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe937d800, cur 1540456736 expire 1540456586 last 1540456509 Oct 25 01:42:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 01:49:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9683c1e5-1f73-2774-9b22-4bfff7c8c3e9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881466ba2800, cur 1540457397 expire 1540457247 last 1540457170 Oct 25 01:54:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e2d30cb4-73b1-b156-156c-5b7926d96fe0 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fef101400, cur 1540457682 expire 1540457532 last 1540457455 Oct 25 01:56:36 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 25 02:01:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 02:22:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e7a5eef4-2c97-ff82-aaa2-87dc338f81df (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817ed251000, cur 1540459331 expire 1540459181 last 1540459104 Oct 25 02:31:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 02:55:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 061cb29d-9cd1-d826-75c6-b4deb8788cd1 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881af2678400, cur 1540461306 expire 1540461156 last 1540461079 Oct 25 02:58:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 25 08:05:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e1e27c55-f0d7-7d9f-c9ca-1201a3792f71 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b28fd9800, cur 1540479948 expire 1540479798 last 1540479721 Oct 25 08:35:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e9c5b478-8f06-f685-1543-48254888848a (at 10.8.15.2@o2ib6) Oct 25 08:35:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 046b6f7b-5531-8a41-d540-00485b286155 (at 10.8.15.3@o2ib6) Oct 25 08:35:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 25 08:37:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 83ca34e7-4cfe-edc7-6cf8-715bf4bb75cd (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f227c8800, cur 1540481834 expire 1540481684 last 1540481607 Oct 25 08:40:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8e6255b2-c53d-43fa-e3b1-899cb60f40be (at 10.12.5.41@o2ib) Oct 25 08:40:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bcb82d82-95e9-a8e8-118a-6a1ad356c48b (at 10.12.5.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f80400, cur 1540482051 expire 1540481901 last 1540481824 Oct 25 09:13:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 196b0fa5-e8c1-02d7-e875-b80a1a44199a (at 10.12.4.33@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88082edfd400, cur 1540483991 expire 1540483841 last 1540483764 Oct 25 09:15:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5a524ba4-15c7-1347-2921-cd73173ea987 (at 10.12.4.33@o2ib) Oct 25 09:49:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client 8c4da535-cfa1-b985-d9ca-f4071a73ae60 (at 10.8.21.3@o2ib6) reconnecting Oct 25 09:49:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 80f771f1-49da-2bd2-19b0-e53fa2e7702c (at 10.8.21.3@o2ib6) Oct 25 10:17:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8e6255b2-c53d-43fa-e3b1-899cb60f40be (at 10.12.5.41@o2ib) Oct 25 10:17:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 0ed9afd8-4c61-b172-a00f-03e9c88e31b6 (at 10.12.5.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bf657e800, cur 1540487868 expire 1540487718 last 1540487641 Oct 25 10:22:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 93bfa03a-8f2f-2db0-9ece-08f0fe4c2307 (at 10.8.4.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88055c14c400, cur 1540488155 expire 1540488005 last 1540487928 Oct 25 10:23:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c4b119af-2df3-64a1-a0dc-9e0c140c1823 (at 10.8.4.12@o2ib6) Oct 25 10:56:24 oak-md1-s2 kernel: LNet: 124103:0:(o2iblnd_cb.c:2405:kiblnd_passive_connect()) Conn stale 10.0.2.223@o2ib5 version 12/12 incarnation 1536166661792320/1540490182609104 Oct 25 10:56:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 07da5e87-6a86-7fdf-7693-c2ca350af724 (at 10.0.2.223@o2ib5) Oct 25 12:38:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5228c5a7-eb77-96f5-1a27-5a3b661fd302 (at 10.9.104.2@o2ib4) Oct 25 12:38:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2ce6fa59-5e31-71ae-af70-fd0bf18fb38d (at 10.9.104.4@o2ib4) Oct 25 12:39:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to b5bece9f-1994-cc1e-a006-b95a2b0d4482 (at 10.9.104.1@o2ib4) Oct 25 18:43:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 12f508f1-2aac-c109-5a37-14a77426c399 (at 10.12.4.47@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805d4fc9400, cur 1540518188 expire 1540518038 last 1540517961 Oct 25 18:43:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to bd969ee3-6566-f7ca-f70b-37bb475a0699 (at 10.12.5.45@o2ib) Oct 25 18:52:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e8119a61-6f86-11ca-8d43-0b09c9ff37fc (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf0e0400, cur 1540518735 expire 1540518585 last 1540518508 Oct 25 18:53:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5395f800-fc74-b932-7b18-dededafb6123 (at 10.8.10.33@o2ib6) Oct 25 18:54:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 410025aa-6f42-c42b-9e79-bce2018737f4 (at 10.12.4.67@o2ib) Oct 25 18:58:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7e92711d-40ae-081c-2fad-5846bf39eae8 (at 10.12.4.67@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039232800, cur 1540519082 expire 1540518932 last 1540518855 Oct 25 19:03:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 410025aa-6f42-c42b-9e79-bce2018737f4 (at 10.12.4.67@o2ib) Oct 25 19:25:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8055fac6-db03-2856-4127-fb53e406bdb2 (at 10.12.5.46@o2ib) Oct 25 19:30:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27bac3ac-d257-a563-4125-c911a485eb0b (at 10.9.101.60@o2ib4) Oct 25 19:34:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8055fac6-db03-2856-4127-fb53e406bdb2 (at 10.12.5.46@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea36fec00, cur 1540521281 expire 1540521131 last 1540521054 Oct 25 19:34:41 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 25 20:43:45 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 19200b8b-cc86-e0d2-0b70-fe218614d10a (at 10.12.5.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8804bc21f000, cur 1540525425 expire 1540525275 last 1540525198 Oct 25 20:45:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9e4987ac-b356-c20f-d1b6-aba8b64918f4 (at 10.12.5.37@o2ib) in 226 seconds. I think it's dead, and I am evicting it. exp ffff8805a6169000, cur 1540525501 expire 1540525351 last 1540525275 Oct 25 20:45:01 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 25 20:45:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 25 20:53:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 34a25f80-7263-29af-b47a-e1d4c4b2d08a (at 10.12.5.40@o2ib) Oct 25 20:53:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a9b40250-5258-6ff9-b055-52c7475dca23 (at 10.12.5.38@o2ib) Oct 25 20:53:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8e6255b2-c53d-43fa-e3b1-899cb60f40be (at 10.12.5.41@o2ib) Oct 25 20:53:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 25 20:53:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a97ac5b8-979e-0386-d35c-c33acc053436 (at 10.12.5.43@o2ib) Oct 25 20:53:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 919dfb2a-ca78-86a4-4022-cc063b1d1e79 (at 10.12.4.23@o2ib) Oct 25 20:53:39 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 25 20:56:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client da5deffa-800b-10c7-8580-a60c48cc7735 (at 10.12.4.23@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b99264c00, cur 1540526187 expire 1540526037 last 1540525960 Oct 25 20:58:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 20696ab3-77f6-e423-0a18-5be4dca7c662 (at 10.12.5.44@o2ib) Oct 25 21:51:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client ab423426-374f-a76d-90fb-a4109d341f8e (at 10.8.7.33@o2ib6) reconnecting Oct 25 21:51:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 17643ea1-88c6-4b8d-444c-581924b9f1d3 (at 10.8.7.33@o2ib6) Oct 25 21:51:11 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 25 22:41:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 27bac3ac-d257-a563-4125-c911a485eb0b (at 10.9.101.60@o2ib4) Oct 25 22:53:35 oak-md1-s2 kernel: LustreError: 7150:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 25 23:41:45 oak-md1-s2 kernel: LustreError: 7164:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 25 23:41:45 oak-md1-s2 kernel: LustreError: 7164:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2 previous similar messages Oct 25 23:45:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 15155f24-c3d3-2a17-893b-50a2b36337a6 (at 10.12.5.38@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880eaa67b800, cur 1540536339 expire 1540536189 last 1540536112 Oct 25 23:47:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 1a61af53-5bf7-48d7-fbff-7a3628c9b074 (at 10.12.5.37@o2ib) Oct 25 23:53:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8e6255b2-c53d-43fa-e3b1-899cb60f40be (at 10.12.5.41@o2ib) Oct 25 23:54:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a97ac5b8-979e-0386-d35c-c33acc053436 (at 10.12.5.43@o2ib) Oct 25 23:54:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 96cb526e-b16d-4292-ee35-19453749e5b6 (at 10.12.5.42@o2ib) Oct 25 23:54:19 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 26 00:48:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ea2908fc-d3a8-bdab-77a4-838ca9b20eea (at 10.8.17.28@o2ib6) Oct 26 00:48:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 00:49:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 138ad3f6-2ee7-4513-c7c8-b9a0c960ba9b (at 10.8.17.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805a616a400, cur 1540540144 expire 1540539994 last 1540539917 Oct 26 00:49:04 oak-md1-s2 kernel: Lustre: Skipped 8 previous similar messages Oct 26 03:44:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Client d32082a0-3e1b-c911-1a50-22de17fb65a2 (at 10.8.27.22@o2ib6) reconnecting Oct 26 03:44:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to f7311cb7-4a52-7fe4-7e17-432ae5dfbf4b (at 10.8.27.22@o2ib6) Oct 26 05:02:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 05:03:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bbaf69be-5bd6-e66d-4785-e8ffb4691b00 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881851233000, cur 1540555385 expire 1540555235 last 1540555158 Oct 26 05:28:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 05:29:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fe304f59-49a2-8356-3ff1-2caa63ece850 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881eccecf000, cur 1540556983 expire 1540556833 last 1540556756 Oct 26 05:40:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2924d241-c237-342a-25d4-6186b676ce8f (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8817a83aa400, cur 1540557658 expire 1540557508 last 1540557431 Oct 26 05:42:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 05:57:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6f8cd523-a7bd-7b12-e253-97c76b68e02c (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881039234400, cur 1540558664 expire 1540558514 last 1540558437 Oct 26 05:57:58 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:10:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d8b8ef6c-f334-5b3f-a8eb-2530c1492464 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88030ca4c000, cur 1540559405 expire 1540559255 last 1540559178 Oct 26 06:10:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:18:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:19:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ef296941-2b7d-5be0-de13-4f68c87f2997 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beb1e0800, cur 1540559946 expire 1540559796 last 1540559719 Oct 26 06:24:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:25:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1f72f5d3-31ba-7170-3442-376f41c1e6d6 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88014e259000, cur 1540560359 expire 1540560209 last 1540560132 Oct 26 06:32:15 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:33:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client dcac30ec-8a57-ad3b-3183-9a178a54a7da (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d773e2800, cur 1540560789 expire 1540560639 last 1540560562 Oct 26 06:39:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:40:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client bf84a3d3-2dba-1106-3b8d-463dab506cbb (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8814b268ec00, cur 1540561212 expire 1540561062 last 1540560985 Oct 26 06:46:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:47:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 05022571-ca74-ac90-97e8-3ef11c8ed577 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8812383b6800, cur 1540561622 expire 1540561472 last 1540561395 Oct 26 06:53:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 06:54:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 04a03696-3ff8-e27f-f9a8-58d1b3aad3fc (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f19683000, cur 1540562068 expire 1540561918 last 1540561841 Oct 26 07:01:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 07:01:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6648aae7-fb41-84e8-a1f6-83dc315b9cb0 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881c29686800, cur 1540562492 expire 1540562342 last 1540562265 Oct 26 07:07:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 07:09:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c938dadc-2834-1199-bac5-3101387f7383 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815972e7c00, cur 1540562942 expire 1540562792 last 1540562715 Oct 26 07:29:02 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 07:29:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2621f798-4f0e-56b1-3bb7-91b3f4e23ec6 (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881999306000, cur 1540564174 expire 1540564024 last 1540563947 Oct 26 07:41:57 oak-md1-s2 kernel: LustreError: 7158:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 07:41:57 oak-md1-s2 kernel: LustreError: 7158:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2 previous similar messages Oct 26 07:41:57 oak-md1-s2 kernel: LustreError: 4271:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 07:41:57 oak-md1-s2 kernel: LustreError: 4271:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 8 previous similar messages Oct 26 07:41:58 oak-md1-s2 kernel: LustreError: 7158:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 07:41:58 oak-md1-s2 kernel: LustreError: 7158:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 9 previous similar messages Oct 26 07:42:01 oak-md1-s2 kernel: LustreError: 4284:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 07:42:01 oak-md1-s2 kernel: LustreError: 4284:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 16 previous similar messages Oct 26 08:39:05 oak-md1-s2 kernel: LustreError: 7161:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 08:39:05 oak-md1-s2 kernel: LustreError: 7161:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2 previous similar messages Oct 26 08:45:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aa675f90-c2fe-0ab6-3178-fab3e374d618 (at 10.9.0.1@o2ib4) Oct 26 09:08:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9909923d-f14d-17cb-0644-65ec18b4c40d (at 10.8.27.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d23000, cur 1540570120 expire 1540569970 last 1540569893 Oct 26 09:09:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to c075bc0a-760c-5dfb-d8e2-b4ad07dbe332 (at 10.8.27.12@o2ib6) Oct 26 09:09:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to a6a36316-7e69-4d65-5bf9-921274477a15 (at 10.8.15.7@o2ib6) Oct 26 10:09:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Oct 26 10:10:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 85e164ed-7676-d8f9-aaa6-51531d36ce45 (at 10.9.101.59@o2ib4) Oct 26 10:16:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to e392ccdf-2ce5-80ab-687b-1efcb7434061 (at 10.9.103.25@o2ib4) Oct 26 10:18:07 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ddcef68-1842-f4c2-fe0b-893e20489be1 (at 10.12.4.63@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b58f82000, cur 1540574287 expire 1540574137 last 1540574060 Oct 26 10:52:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 48ec7f74-5b7c-f0ad-fb35-6041a89f83d3 (at 10.8.4.16@o2ib6) Oct 26 11:09:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to aa675f90-c2fe-0ab6-3178-fab3e374d618 (at 10.9.0.1@o2ib4) Oct 26 11:13:19 oak-md1-s2 kernel: LustreError: 7146:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 11:17:56 oak-md1-s2 kernel: LustreError: 7158:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 11:17:56 oak-md1-s2 kernel: LustreError: 7158:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2 previous similar messages Oct 26 11:18:58 oak-md1-s2 kernel: Lustre: 10633:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540577931/real 1540577931] req@ffff8802d1bec200 x1610636037040592/t0(0) o104->oak-MDT0000@10.9.112.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540577938 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 26 11:18:58 oak-md1-s2 kernel: Lustre: 10633:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 55 previous similar messages Oct 26 11:20:15 oak-md1-s2 kernel: Lustre: 10633:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540578008/real 1540578008] req@ffff8802d1bec200 x1610636037040592/t0(0) o104->oak-MDT0000@10.9.112.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540578015 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 26 11:20:15 oak-md1-s2 kernel: Lustre: 10633:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Oct 26 11:20:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 9facc8bb-08ef-d05c-acb9-a9dd9992070d (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881959695000, cur 1540578054 expire 1540577904 last 1540577827 Oct 26 11:21:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 8bf97987-9db5-9366-1066-f838f72f4da1 (at 10.9.112.3@o2ib4) Oct 26 11:37:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b79a8fd6-a502-801d-637e-e6e652109c54 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805f1811400, cur 1540579054 expire 1540578904 last 1540578827 Oct 26 11:42:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 11:49:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 11:50:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 622a6c89-58cb-8a78-318c-1a991ebd2441 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe9378000, cur 1540579806 expire 1540579656 last 1540579579 Oct 26 11:57:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7b90179c-087a-cb97-6d8f-20715056475b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880590d6f800, cur 1540580226 expire 1540580076 last 1540579999 Oct 26 11:58:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:01:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 46cdf3ac-3772-4a22-e0d2-34d9f0963c1f (at 10.8.9.5@o2ib6) Oct 26 12:04:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:05:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fc559ee3-67f7-cebc-ba21-a41ccbdcf838 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881da8ef0000, cur 1540580732 expire 1540580582 last 1540580505 Oct 26 12:11:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e62d39b1-0a76-7601-7c00-dc4284fb9937 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d21343000, cur 1540581076 expire 1540580926 last 1540580849 Oct 26 12:13:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:21:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:22:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client b6d5430e-a637-3e0d-a8e2-7e7eefde1152 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803ee803800, cur 1540581768 expire 1540581618 last 1540581541 Oct 26 12:28:18 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to ca7e808c-fc24-14c5-a11f-62ad1ed8a1f2 (at 10.9.103.26@o2ib4) Oct 26 12:28:18 oak-md1-s2 kernel: Lustre: Skipped 7 previous similar messages Oct 26 12:31:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:31:39 oak-md1-s2 kernel: Lustre: Skipped 3 previous similar messages Oct 26 12:32:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 79493c8d-3469-2578-5007-f858b19f2b88 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803cb636c00, cur 1540582336 expire 1540582186 last 1540582109 Oct 26 12:38:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 10921b42-e99c-7f95-6aa3-bec0282ebc34 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8815972e0800, cur 1540582702 expire 1540582552 last 1540582475 Oct 26 12:38:22 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 26 12:38:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:45:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:46:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ee839ce5-2b4c-48d7-b2c9-4023cb902c3d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803352b1000, cur 1540583191 expire 1540583041 last 1540582964 Oct 26 12:52:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 713f7a2d-5b1b-78fd-1592-533a852b5d6e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8802955b3800, cur 1540583559 expire 1540583409 last 1540583332 Oct 26 12:54:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 12:54:09 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 26 13:01:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f56adb7f-245f-56d5-e16f-dcb1056fee9e (at 10.9.105.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881037d46000, cur 1540584063 expire 1540583913 last 1540583836 Oct 26 13:02:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 4e77b2b7-e777-953e-022f-51b17f02f081 (at 10.8.20.15@o2ib6) in 165 seconds. I think it's dead, and I am evicting it. exp ffff8814f774c800, cur 1540584139 expire 1540583989 last 1540583974 Oct 26 13:04:11 oak-md1-s2 kernel: Lustre: 10500:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540584244/real 1540584244] req@ffff881f5e9bbc00 x1610636059555568/t0(0) o104->oak-MDT0000@10.9.105.50@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540584251 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 26 13:04:11 oak-md1-s2 kernel: Lustre: 10500:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 26 13:04:32 oak-md1-s2 kernel: Lustre: 10500:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540584265/real 1540584265] req@ffff881f5e9bbc00 x1610636059555568/t0(0) o104->oak-MDT0000@10.9.105.50@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540584272 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 26 13:04:32 oak-md1-s2 kernel: Lustre: 10500:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 26 13:05:15 oak-md1-s2 kernel: Lustre: 10500:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540584307/real 1540584307] req@ffff881f5e9bbc00 x1610636059555568/t0(0) o104->oak-MDT0000@10.9.105.50@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1540584314 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 26 13:05:15 oak-md1-s2 kernel: Lustre: 10500:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 26 13:05:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 13:05:42 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 13:05:50 oak-md1-s2 kernel: LustreError: 10500:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.105.50@o2ib4) failed to reply to blocking AST (req@ffff881f5e9bbc00 x1610636059555568 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff881442b97600/0x6c00c44cf8903cc0 lrc: 4/0,0 mode: PR/PR res: [0x2000181e8:0xac10:0x0].0x0 bits 0x13 rrc: 160 type: IBT flags: 0x60200400000020 nid: 10.9.105.50@o2ib4 remote: 0xf3941a9392668344 expref: 31 pid: 10535 timeout: 8868636104 lvb_type: 0 Oct 26 13:05:50 oak-md1-s2 kernel: LustreError: 138-a: oak-MDT0000: A client on nid 10.9.105.50@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Oct 26 13:12:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1b1fc0ac-c7f1-827c-20ec-82c3095d601b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8800264a6400, cur 1540584744 expire 1540584594 last 1540584517 Oct 26 13:21:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 13:21:11 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 26 13:21:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 02938050-6e1b-8790-4b89-5933b1bfdef3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daccb6400, cur 1540585304 expire 1540585154 last 1540585077 Oct 26 13:24:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3666eb1c-762d-2b11-5547-317c7c779c13 (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880beea0a800, cur 1540585467 expire 1540585317 last 1540585240 Oct 26 13:33:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 13:33:14 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 26 13:34:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a6c3ce44-1ed2-8960-ad62-1f2758121a7b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88076f78b800, cur 1540586045 expire 1540585895 last 1540585818 Oct 26 13:34:05 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 13:57:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 72097c8d-6b91-49cf-589c-86d2d5b9e08f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88031f765800, cur 1540587471 expire 1540587321 last 1540587244 Oct 26 14:13:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 271597b8-7305-5b82-15d7-58d2b2a4dd26 (at 10.9.104.3@o2ib4) Oct 26 14:15:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 2f6ba610-8421-b332-c91e-1efe29c7e7dd (at 10.8.15.1@o2ib6) Oct 26 14:15:21 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 14:21:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 14:36:52 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540589805/real 1540589805] req@ffff88051de51800 x1610636064815280/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540589812 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Oct 26 14:36:52 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 26 14:37:06 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540589819/real 1540589819] req@ffff88051de51800 x1610636064815280/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540589826 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 26 14:37:06 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Oct 26 14:37:27 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540589840/real 1540589840] req@ffff88051de51800 x1610636064815280/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540589847 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 26 14:37:27 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Oct 26 14:38:09 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1540589882/real 1540589882] req@ffff88051de51800 x1610636064815280/t0(0) o104->oak-MDT0000@10.8.17.27@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1540589889 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Oct 26 14:38:09 oak-md1-s2 kernel: Lustre: 10508:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Oct 26 14:39:11 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e17cc8fc-bd26-f7a0-ff23-9ce0b84e0d42 (at 10.8.17.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b95224c00, cur 1540589951 expire 1540589801 last 1540589724 Oct 26 14:40:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 26 14:45:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 510c3bf4-24ab-7dd0-d57e-e4f867a5cae8 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805f1810800, cur 1540590303 expire 1540590153 last 1540590076 Oct 26 14:45:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 5dbe74c0-0cd4-fbb4-0f82-120be78b935a (at 10.8.4.15@o2ib6) Oct 26 14:57:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 256a544b-6980-5a97-75ef-a13c9d76dad4 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88005b9a2000, cur 1540591046 expire 1540590896 last 1540590819 Oct 26 15:04:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 59a22b4d-b5c2-30ef-e068-247345284d26 (at 10.8.17.27@o2ib6) Oct 26 15:14:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client e94e4d76-efee-120c-2890-a005db8ae121 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8805f1813800, cur 1540592065 expire 1540591915 last 1540591838 Oct 26 15:38:17 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 297bcea1-55ef-8f29-66b7-67c5ab05a8c6 (at 10.9.112.15@o2ib4) Oct 26 15:39:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 15:46:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 15:46:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 298ff4f7-ec33-7c4e-60e7-b7a0de61f280 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8809439dec00, cur 1540593976 expire 1540593826 last 1540593749 Oct 26 15:49:40 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 15:50:27 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 1ff69a30-fbb8-2aef-c415-083e075291eb (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e13937c00, cur 1540594227 expire 1540594077 last 1540594000 Oct 26 15:51:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac7a31dd-efde-f651-9701-b38847a6d278 (at 10.12.5.39@o2ib) in 220 seconds. I think it's dead, and I am evicting it. exp ffff880004431400, cur 1540594303 expire 1540594153 last 1540594083 Oct 26 15:52:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 67533ae6-18d9-00cf-74a0-166d6fc6ab0a (at 10.12.5.39@o2ib) Oct 26 15:52:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ac45206e-2ec2-fbe6-50f5-68404ac9353d (at 10.8.20.19@o2ib6) in 165 seconds. I think it's dead, and I am evicting it. exp ffff881fe534a400, cur 1540594379 expire 1540594229 last 1540594214 Oct 26 15:55:35 oak-md1-s2 kernel: Lustre: 10605:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 15:55:35 oak-md1-s2 kernel: Lustre: 10605:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 40 previous similar messages Oct 26 15:56:09 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 15:56:09 oak-md1-s2 kernel: Lustre: 10496:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages Oct 26 15:56:51 oak-md1-s2 kernel: Lustre: 10491:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 15:56:51 oak-md1-s2 kernel: Lustre: 10491:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 145 previous similar messages Oct 26 15:57:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 58dd2e14-adee-7e3e-b574-eda896b5facb (at 10.8.7.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f6cc6cc00, cur 1540594651 expire 1540594501 last 1540594424 Oct 26 15:57:45 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 15:57:45 oak-md1-s2 kernel: Lustre: 10545:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages Oct 26 15:59:21 oak-md1-s2 kernel: Lustre: 10511:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 15:59:21 oak-md1-s2 kernel: Lustre: 10511:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 345 previous similar messages Oct 26 15:59:38 oak-md1-s2 kernel: Lustre: 10583:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 16:00:10 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 16:00:10 oak-md1-s2 kernel: Lustre: 10565:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages Oct 26 16:00:56 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 16:00:56 oak-md1-s2 kernel: Lustre: 4256:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages Oct 26 16:02:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 16:02:04 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 16:02:22 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 16:02:22 oak-md1-s2 kernel: Lustre: 10651:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 46 previous similar messages Oct 26 16:02:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a806cbc0-ed6b-9197-c595-a89681f95aea (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881fe5a59800, cur 1540594969 expire 1540594819 last 1540594742 Oct 26 16:04:22 oak-md1-s2 kernel: Lustre: 4276:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 16:04:22 oak-md1-s2 kernel: Lustre: 4276:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 621 previous similar messages Oct 26 16:05:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 16:06:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client ad49ff3d-4a22-4f0f-3399-5a3dba7dd1c8 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88195c6bf000, cur 1540595183 expire 1540595033 last 1540594956 Oct 26 16:13:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 16:14:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client d3f38c5b-313f-bc62-7cdb-0e9f128a7cc6 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880d0947c400, cur 1540595649 expire 1540595499 last 1540595422 Oct 26 16:14:24 oak-md1-s2 kernel: Lustre: 10545:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 16:14:24 oak-md1-s2 kernel: Lustre: 10545:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 784 previous similar messages Oct 26 16:19:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 16:20:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8cb69785-02c8-2b2e-fdef-c36ad191415f (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8803bbafb000, cur 1540596000 expire 1540595850 last 1540595773 Oct 26 16:23:38 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 16:23:38 oak-md1-s2 kernel: Lustre: 10639:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages Oct 26 16:24:24 oak-md1-s2 kernel: Lustre: 4303:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 16:24:24 oak-md1-s2 kernel: Lustre: 4303:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1103 previous similar messages Oct 26 16:24:43 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 626a6b87-b14d-8460-9aa3-a4e226e243e0 (at 10.8.7.9@o2ib6) Oct 26 16:31:21 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 49ecba36-9768-f8ed-0e63-e860b7f06095 (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88001c0fd000, cur 1540596681 expire 1540596531 last 1540596454 Oct 26 16:34:25 oak-md1-s2 kernel: Lustre: 10551:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 16:34:25 oak-md1-s2 kernel: Lustre: 10551:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1347 previous similar messages Oct 26 16:44:25 oak-md1-s2 kernel: Lustre: 10542:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 16:44:25 oak-md1-s2 kernel: Lustre: 10542:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1034 previous similar messages Oct 26 16:48:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 42187a54-186f-902f-18d9-57d1a0fc71d3 (at 10.8.20.15@o2ib6) Oct 26 16:49:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 6c60e781-20f9-f18a-83a7-dc62d2001411 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ea512a800, cur 1540597774 expire 1540597624 last 1540597547 Oct 26 16:50:21 oak-md1-s2 kernel: Lustre: 10652:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 16:54:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 2bab0b81-62e6-f538-22ae-769491631542 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e25d26400, cur 1540598048 expire 1540597898 last 1540597821 Oct 26 16:54:25 oak-md1-s2 kernel: Lustre: 10551:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 16:54:25 oak-md1-s2 kernel: Lustre: 10551:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 821 previous similar messages Oct 26 16:54:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 16:57:59 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to df15e742-4eff-75f9-e69e-14044c7130de (at 10.8.20.19@o2ib6) Oct 26 16:57:59 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 16:59:00 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5fc54850-a09a-24d1-86ea-5b7bf84c844d (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880954f8fc00, cur 1540598340 expire 1540598190 last 1540598113 Oct 26 17:01:46 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c983e43b-3838-e165-0211-8b142aed89c9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880052c84800, cur 1540598506 expire 1540598356 last 1540598279 Oct 26 17:04:25 oak-md1-s2 kernel: Lustre: 10529:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 17:04:25 oak-md1-s2 kernel: Lustre: 10529:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 769 previous similar messages Oct 26 17:14:26 oak-md1-s2 kernel: Lustre: 4694:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 17:14:26 oak-md1-s2 kernel: Lustre: 4694:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 415 previous similar messages Oct 26 17:24:26 oak-md1-s2 kernel: Lustre: 10644:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 17:24:26 oak-md1-s2 kernel: Lustre: 10644:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 334 previous similar messages Oct 26 17:26:13 oak-md1-s2 kernel: LDISKFS-fs error (device dm-0): ldiskfs_map_blocks:594: inode #659619751: block 774843950: comm mdt00_100: lblock 0 mapped to illegal pblock (length 1) Oct 26 17:26:13 oak-md1-s2 kernel: Aborting journal on device dm-0-8. Oct 26 17:26:13 oak-md1-s2 kernel: LustreError: 3844:0:(osd_handler.c:1586:osd_trans_commit_cb()) transaction @0xffff881b024c6a80 commit error: 2 Oct 26 17:26:13 oak-md1-s2 kernel: LDISKFS-fs (dm-0): Remounting filesystem read-only Oct 26 17:26:13 oak-md1-s2 kernel: LustreError: 10644:0:(osd_io.c:1494:osd_ldiskfs_read()) dm-0: can't read 59@0 on ino 659619751: rc = -5 Oct 26 17:26:13 oak-md1-s2 kernel: LustreError: 10644:0:(mdd_dir.c:3525:mdd_migrate_create()) oak-MDD0000: [0x200002b64:0x1335e:0x0] readlink failed: rc = -5 Oct 26 17:26:13 oak-md1-s2 kernel: LDISKFS-fs error (device dm-0): ldiskfs_map_blocks:594: inode #659619748: block 774843950: comm mdt00_111: lblock 0 mapped to illegal pblock (length 1) Oct 26 17:26:13 oak-md1-s2 kernel: LustreError: 4011:0:(llog_cat.c:795:llog_cat_cancel_records()) oak-OST0037-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -30 Oct 26 17:26:13 oak-md1-s2 kernel: LustreError: 4011:0:(osp_sync.c:1060:osp_sync_process_committed()) oak-OST0037-osc-MDT0000: can't cancel record: -30 Oct 26 17:26:14 oak-md1-s2 kernel: LustreError: 7167:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x1056:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:3623 enforced:1 hard:40000000000 soft:40000000000 granted:38748159680 time:0 qunit:4194304 edquot:0 may_rel:0 revoke:0 Oct 26 17:26:14 oak-md1-s2 kernel: LustreError: 4057:0:(llog_cat.c:795:llog_cat_cancel_records()) oak-OST004e-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -30 Oct 26 17:26:14 oak-md1-s2 kernel: LustreError: 4057:0:(llog_cat.c:795:llog_cat_cancel_records()) Skipped 5 previous similar messages Oct 26 17:26:14 oak-md1-s2 kernel: LustreError: 4057:0:(osp_sync.c:1060:osp_sync_process_committed()) oak-OST004e-osc-MDT0000: can't cancel record: -30 Oct 26 17:26:14 oak-md1-s2 kernel: LustreError: 4057:0:(osp_sync.c:1060:osp_sync_process_committed()) Skipped 1 previous similar message Oct 26 17:26:14 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:26:14 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message Oct 26 17:26:15 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:26:15 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 162 previous similar messages Oct 26 17:26:15 oak-md1-s2 kernel: LustreError: 4087:0:(llog_cat.c:795:llog_cat_cancel_records()) oak-OST005d-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -30 Oct 26 17:26:15 oak-md1-s2 kernel: LustreError: 4073:0:(osp_sync.c:1060:osp_sync_process_committed()) oak-OST0056-osc-MDT0000: can't cancel record: -30 Oct 26 17:26:15 oak-md1-s2 kernel: LustreError: 4073:0:(osp_sync.c:1060:osp_sync_process_committed()) Skipped 17 previous similar messages Oct 26 17:26:15 oak-md1-s2 kernel: LustreError: 4087:0:(llog_cat.c:795:llog_cat_cancel_records()) Skipped 20 previous similar messages Oct 26 17:26:16 oak-md1-s2 kernel: Lustre: 10566:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:26:16 oak-md1-s2 kernel: Lustre: 10566:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 419 previous similar messages Oct 26 17:26:16 oak-md1-s2 kernel: LDISKFS-fs warning (device dm-0): kmmpd:187: kmmpd being stopped since filesystem has been remounted as readonly. Oct 26 17:26:17 oak-md1-s2 kernel: LustreError: 3876:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x1033:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:5250 enforced:1 hard:10000000000 soft:10000000000 granted:8752904128 time:0 qunit:4194304 edquot:0 may_rel:0 revoke:0 Oct 26 17:26:17 oak-md1-s2 kernel: LustreError: 3949:0:(llog_cat.c:795:llog_cat_cancel_records()) oak-OST0018-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -30 Oct 26 17:26:17 oak-md1-s2 kernel: LustreError: 3949:0:(llog_cat.c:795:llog_cat_cancel_records()) Skipped 35 previous similar messages Oct 26 17:26:17 oak-md1-s2 kernel: LustreError: 3949:0:(osp_sync.c:1060:osp_sync_process_committed()) oak-OST0018-osc-MDT0000: can't cancel record: -30 Oct 26 17:26:17 oak-md1-s2 kernel: LustreError: 3949:0:(osp_sync.c:1060:osp_sync_process_committed()) Skipped 38 previous similar messages Oct 26 17:26:18 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:26:18 oak-md1-s2 kernel: Lustre: 10543:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1241 previous similar messages Oct 26 17:26:21 oak-md1-s2 kernel: LustreError: 7153:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x105a:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:3362 enforced:1 hard:80000000000 soft:80000000000 granted:72593587780 time:0 qunit:16777216 edquot:0 may_rel:0 revoke:0 Oct 26 17:26:22 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:26:22 oak-md1-s2 kernel: Lustre: 10644:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2130 previous similar messages Oct 26 17:26:24 oak-md1-s2 kernel: LustreError: 7167:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x104a:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:3377 enforced:1 hard:250000000000 soft:250000000000 granted:231343896840 time:0 qunit:268435456 edquot:0 may_rel:0 revoke:0 Oct 26 17:26:30 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:26:30 oak-md1-s2 kernel: Lustre: 10653:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5123 previous similar messages Oct 26 17:26:31 oak-md1-s2 kernel: LustreError: 7153:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x1044:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:3281 enforced:1 hard:20000000000 soft:20000000000 granted:6333354480 time:0 qunit:67108864 edquot:0 may_rel:0 revoke:0 Oct 26 17:26:39 oak-md1-s2 kernel: LustreError: 4242:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x1037:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:5381 enforced:1 hard:60000000000 soft:60000000000 granted:29889527012 time:0 qunit:67108864 edquot:0 may_rel:0 revoke:0 Oct 26 17:26:39 oak-md1-s2 kernel: LustreError: 4242:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) Skipped 5 previous similar messages Oct 26 17:26:46 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:26:46 oak-md1-s2 kernel: Lustre: 10549:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8623 previous similar messages Oct 26 17:26:57 oak-md1-s2 kernel: LustreError: 4242:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x1057:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:3281 enforced:1 hard:20000000000 soft:20000000000 granted:6333354480 time:0 qunit:67108864 edquot:0 may_rel:0 revoke:0 Oct 26 17:26:57 oak-md1-s2 kernel: LustreError: 4242:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) Skipped 13 previous similar messages Oct 26 17:27:18 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -30 Oct 26 17:27:18 oak-md1-s2 kernel: Lustre: 10529:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21436 previous similar messages Oct 26 17:27:21 oak-md1-s2 kernel: Lustre: DEBUG MARKER: Fri Oct 26 17:27:21 2018 Oct 26 17:27:29 oak-md1-s2 kernel: LustreError: 4242:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x1037:0x0], rc:-30 qmt:oak-QMT0000 pool:0-dt id:5381 enforced:1 hard:60000000000 soft:60000000000 granted:29889527012 time:0 qunit:67108864 edquot:0 may_rel:0 revoke:0 Oct 26 17:27:29 oak-md1-s2 kernel: LustreError: 4242:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) Skipped 79 previous similar messages Oct 26 17:27:30 oak-md1-s2 kernel: LustreError: 3810:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:27:36 oak-md1-s2 kernel: LustreError: 3808:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:27:36 oak-md1-s2 kernel: LustreError: 3808:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 13 previous similar messages Oct 26 17:27:43 oak-md1-s2 kernel: LustreError: 3811:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:27:43 oak-md1-s2 kernel: LustreError: 3811:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 49 previous similar messages Oct 26 17:27:50 oak-md1-s2 kernel: LustreError: 3803:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:27:50 oak-md1-s2 kernel: LustreError: 3803:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 49 previous similar messages Oct 26 17:27:58 oak-md1-s2 kernel: LustreError: 3811:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:27:58 oak-md1-s2 kernel: LustreError: 3811:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 49 previous similar messages Oct 26 17:28:08 oak-md1-s2 kernel: LustreError: 3808:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:28:08 oak-md1-s2 kernel: LustreError: 3808:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 49 previous similar messages Oct 26 17:28:26 oak-md1-s2 kernel: LustreError: 3814:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:28:26 oak-md1-s2 kernel: LustreError: 3814:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 99 previous similar messages Oct 26 17:28:33 oak-md1-s2 kernel: LustreError: 4257:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) $$$ failed to slv declare write for [0x200000005:0x1007:0x0], rc:-30 qmt:oak-QMT0000 pool:0-md id:3377 enforced:1 hard:37500000 soft:37500000 granted:35402848 time:0 qunit:4194304 edquot:0 may_rel:0 revoke:0 Oct 26 17:28:33 oak-md1-s2 kernel: LustreError: 4257:0:(qmt_entry.c:201:qmt_trans_start_with_slv()) Skipped 508 previous similar messages Oct 26 17:28:58 oak-md1-s2 kernel: LustreError: 3796:0:(qsd_handler.c:340:qsd_req_completion()) $$$ DQACQ failed with -30, flags:0x1 qsd:oak-MDT0000 qtype:grp id:3377 enforced:1 granted:35402848 pending:0 waiting:0 req:1 usage:33729572 qunit:0 qtune:0 edquot:0 Oct 26 17:28:58 oak-md1-s2 kernel: LustreError: 3796:0:(qsd_handler.c:340:qsd_req_completion()) Skipped 272 previous similar messages Oct 26 17:29:16 oak-md1-s2 kernel: LustreError: 7156:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 17:29:16 oak-md1-s2 kernel: LustreError: 7156:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2 previous similar messages Oct 26 17:29:18 oak-md1-s2 kernel: LustreError: 181431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 17:29:20 oak-md1-s2 kernel: LustreError: 181431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 Oct 26 17:29:47 oak-md1-s2 kernel: Lustre: Failing over oak-MDT0000 Oct 26 17:29:47 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.9.106.7@o2ib4 (stopping) Oct 26 17:29:48 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.9.0.61@o2ib4 arrived at 1540600188 with bad export cookie 7782435669529867174 Oct 26 17:29:48 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) ### ### ns: mdt-oak-MDT0000_UUID lock: ffff8801638ab800/0x6c00c44d1bee4717 lrc: 3/0,0 mode: PR/PR res: [0x2000181e7:0x1bc2f:0x0].0x0 bits 0x13 rrc: 2 type: IBT flags: 0x40200000000000 nid: 10.9.0.61@o2ib4 remote: 0xdd9ff76ab8931d2f expref: 1045 pid: 3868 timeout: 0 lvb_type: 0 Oct 26 17:29:48 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.9.102.40@o2ib4 (stopping) Oct 26 17:29:48 oak-md1-s2 kernel: Lustre: Skipped 34 previous similar messages Oct 26 17:29:48 oak-md1-s2 kernel: LustreError: 3796:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88060cdcef00 x1610636110353776/t0(0) o13->oak-OST002c-osc-MDT0000@10.0.2.101@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 26 17:29:48 oak-md1-s2 kernel: LustreError: 3796:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 1 previous similar message Oct 26 17:29:49 oak-md1-s2 kernel: LustreError: 3791:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88175d0ddd00 x1610636110354736/t0(0) o13->oak-OST0042-osc-MDT0000@10.0.2.105@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 26 17:29:49 oak-md1-s2 kernel: LustreError: 3814:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88109f14c500 x1610636110354704/t0(0) o13->oak-OST004a-osc-MDT0000@10.0.2.105@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 26 17:29:49 oak-md1-s2 kernel: LustreError: 3813:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff881d6e196300 x1610636110354720/t0(0) o13->oak-OST001b-osc-MDT0000@10.0.2.102@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 26 17:29:49 oak-md1-s2 kernel: LustreError: 3814:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 9 previous similar messages Oct 26 17:29:49 oak-md1-s2 kernel: LustreError: 3813:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 9 previous similar messages Oct 26 17:29:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.8.17.17@o2ib6 (stopping) Oct 26 17:29:49 oak-md1-s2 kernel: Lustre: Skipped 67 previous similar messages Oct 26 17:29:51 oak-md1-s2 kernel: LustreError: 3793:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8804bac13900 x1610636110355344/t0(0) o13->oak-OST0049-osc-MDT0000@10.0.2.106@o2ib5:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Oct 26 17:29:51 oak-md1-s2 kernel: LustreError: 3793:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 36 previous similar messages Oct 26 17:29:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.8.7.2@o2ib6 (stopping) Oct 26 17:29:51 oak-md1-s2 kernel: Lustre: Skipped 117 previous similar messages Oct 26 17:29:52 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.9.114.13@o2ib4 arrived at 1540600192 with bad export cookie 7782435631684587723 Oct 26 17:29:52 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) ### ### ns: mdt-oak-MDT0000_UUID lock: ffff8801ebb15a00/0x6c00c44d196b74d4 lrc: 3/0,0 mode: PR/PR res: [0x20001806f:0x11c55:0x0].0x0 bits 0x1b rrc: 3 type: IBT flags: 0x40200000000000 nid: 10.9.114.13@o2ib4 remote: 0xbccdbe80feed8cf2 expref: 3963 pid: 10642 timeout: 0 lvb_type: 0 Oct 26 17:29:52 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) Skipped 1 previous similar message Oct 26 17:29:54 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.0.2.105@o2ib5 arrived at 1540600194 with bad export cookie 7782435437524696882 Oct 26 17:29:54 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) Skipped 1 previous similar message Oct 26 17:29:54 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) ### ### ns: mdt-oak-MDT0000_UUID lock: ffff880e36b6f400/0x6c00c44d1bf84c9e lrc: 3/0,0 mode: CR/CR res: [0x200000006:0x1020000:0x1271].0x0 rrc: 142 type: PLN flags: 0x40000000000000 nid: 10.0.2.105@o2ib5 remote: 0x2d5f747e407525f7 expref: 40 pid: 7158 timeout: 0 lvb_type: 2 Oct 26 17:29:55 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.8.1.25@o2ib6 (stopping) Oct 26 17:29:55 oak-md1-s2 kernel: Lustre: Skipped 129 previous similar messages Oct 26 17:29:57 oak-md1-s2 kernel: LustreError: 4872:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.0.2.106@o2ib5 arrived at 1540600196 with bad export cookie 7782435437524697540 Oct 26 17:29:57 oak-md1-s2 kernel: LustreError: 4872:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) Skipped 1 previous similar message Oct 26 17:29:57 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) ### ### ns: mdt-oak-MDT0000_UUID lock: ffff880cc3e66a00/0x6c00c44d1bfaa81f lrc: 3/0,0 mode: PR/PR res: [0x200013665:0x203f:0x0].0x0 bits 0x1b rrc: 2 type: IBT flags: 0x40200000000000 nid: 10.0.2.3@o2ib5 remote: 0x5eb4317026a63345 expref: 1446214 pid: 10551 timeout: 0 lvb_type: 0 Oct 26 17:29:57 oak-md1-s2 kernel: LustreError: 260422:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) Skipped 1 previous similar message Oct 26 17:30:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Not available for connect from 10.8.22.27@o2ib6 (stopping) Oct 26 17:30:03 oak-md1-s2 kernel: Lustre: Skipped 327 previous similar messages Oct 26 17:30:03 oak-md1-s2 kernel: LustreError: 4872:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) ldlm_cancel from 10.8.7.21@o2ib6 arrived at 1540600203 with bad export cookie 7782435648893705481 Oct 26 17:30:03 oak-md1-s2 kernel: LustreError: 4872:0:(ldlm_lockd.c:2365:ldlm_cancel_handler()) Skipped 4 previous similar messages Oct 26 17:30:03 oak-md1-s2 kernel: LustreError: 10852:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) ### ### ns: mdt-oak-MDT0000_UUID lock: ffff8819e1bb6200/0x6c00c44d1c0716da lrc: 3/0,0 mode: CR/CR res: [0x200000006:0x1020000:0x16f9].0x0 rrc: 58 type: PLN flags: 0x40000000000000 nid: 10.0.2.101@o2ib5 remote: 0x12cff173a877367b expref: 75 pid: 4288 timeout: 0 lvb_type: 2 Oct 26 17:30:03 oak-md1-s2 kernel: LustreError: 10852:0:(ldlm_lock.c:2718:ldlm_lock_dump_handle()) Skipped 1 previous similar message Oct 26 17:30:13 oak-md1-s2 kernel: LustreError: 34985:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88109f148c00 x1610636110356176/t0(0) o101->oak-MDT0000-lwp-MDT0000@0@lo:23/10 lens 456/496 e 0 to 0 dl 0 ref 2 fl Rpc:/0/ffffffff rc 0/-1 Oct 26 17:30:13 oak-md1-s2 kernel: LustreError: 34869:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:oak-MDT0000 qtype:grp id:5945 enforced:1 granted:7864321 pending:0 waiting:0 req:0 usage:7287248 qunit:0 qtune:0 edquot:0 Oct 26 17:30:13 oak-md1-s2 kernel: LustreError: 34985:0:(client.c:1166:ptlrpc_import_delay_req()) Skipped 49 previous similar messages Oct 26 17:30:13 oak-md1-s2 kernel: LustreError: 34985:0:(qsd_reint.c:56:qsd_reint_completion()) oak-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x1010000:0x0], rc:-5 Oct 26 17:30:14 oak-md1-s2 kernel: LustreError: 34869:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:oak-MDT0000 qtype:grp id:5945 enforced:1 granted:7864321 pending:0 waiting:0 req:0 usage:7287248 qunit:0 qtune:0 edquot:0 Oct 26 17:30:14 oak-md1-s2 kernel: LustreError: 34869:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 63245 previous similar messages Oct 26 17:30:15 oak-md1-s2 kernel: LustreError: 34869:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage, rc:-3 qsd:oak-MDT0000 qtype:grp id:5945 enforced:1 granted:7864321 pending:0 waiting:0 req:0 usage:7287248 qunit:0 qtune:0 edquot:0 Oct 26 17:30:15 oak-md1-s2 kernel: LustreError: 34869:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 114729 previous similar messages Oct 26 17:30:15 oak-md1-s2 kernel: LustreError: 3911:0:(llog_cat.c:318:llog_cat_close()) oak-OST0000-osc-MDT0000: failure destroying log during cleanup: rc = -30 Oct 26 17:30:15 oak-md1-s2 kernel: Lustre: server umount oak-MDT0000 complete Oct 26 17:30:38 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.210@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:30:38 oak-md1-s2 kernel: LNet: 35147:0:(o2iblnd.c:3019:kiblnd_shutdown()) 10.0.2.52@o2ib5: waiting for 1 peers to disconnect Oct 26 17:30:39 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.210@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:30:39 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 2 previous similar messages Oct 26 17:30:40 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.204@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:30:40 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 3 previous similar messages Oct 26 17:30:42 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.210@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:30:42 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 2 previous similar messages Oct 26 17:30:42 oak-md1-s2 kernel: LNet: 35147:0:(o2iblnd.c:3019:kiblnd_shutdown()) 10.0.2.52@o2ib5: waiting for 1 peers to disconnect Oct 26 17:30:46 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.210@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:30:46 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 5 previous similar messages Oct 26 17:30:51 oak-md1-s2 kernel: LNet: 35147:0:(o2iblnd.c:3019:kiblnd_shutdown()) 10.0.2.52@o2ib5: waiting for 1 peers to disconnect Oct 26 17:30:55 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.210@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:30:55 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 13 previous similar messages Oct 26 17:31:07 oak-md1-s2 kernel: LNet: 35147:0:(o2iblnd.c:3019:kiblnd_shutdown()) 10.0.2.52@o2ib5: waiting for 1 peers to disconnect Oct 26 17:31:11 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.210@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:31:11 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 65 previous similar messages Oct 26 17:31:39 oak-md1-s2 kernel: LNet: 35147:0:(o2iblnd.c:3019:kiblnd_shutdown()) 10.0.2.52@o2ib5: waiting for 1 peers to disconnect Oct 26 17:31:43 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.201@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 17:31:43 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Skipped 91 previous similar messages Oct 26 17:31:45 oak-md1-s2 kernel: LNet: Removed LNI 10.0.2.52@o2ib5 Oct 26 17:31:45 oak-md1-s2 kernel: LustreError: 35213:0:(class_obd.c:701:obdclass_exit()) obd_memory max: 26747623573, leaked: 48392 Oct 26 18:38:55 oak-md1-s2 kernel: LNet: HW NUMA nodes: 2, HW CPU cores: 24, npartitions: 2 Oct 26 18:38:55 oak-md1-s2 kernel: alg: No test for adler32 (adler32-zlib) Oct 26 18:38:55 oak-md1-s2 kernel: alg: No test for crc32 (crc32-table) Oct 26 18:38:56 oak-md1-s2 kernel: Lustre: Lustre: Build Version: 2.10.4 Oct 26 18:38:56 oak-md1-s2 kernel: LNet: Using FMR for registration Oct 26 18:38:56 oak-md1-s2 kernel: LNet: Added LNI 10.0.2.52@o2ib5 [8/256/0/180] Oct 26 18:38:56 oak-md1-s2 kernel: LNetError: 124103:0:(o2iblnd_cb.c:2299:kiblnd_passive_connect()) Can't accept conn from 10.0.2.224@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 Oct 26 18:39:01 oak-md1-s2 kernel: LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc Oct 26 18:39:05 oak-md1-s2 kernel: Lustre: oak-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Oct 26 18:39:05 oak-md1-s2 kernel: Lustre: 36112:0:(llog_cat.c:1061:llog_cat_reverse_process()) oak-MDD0000: catalog [0x7:0xa:0x0] crosses index zero Oct 26 18:39:06 oak-md1-s2 kernel: Lustre: oak-MDD0000: changelog on Oct 26 18:39:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: in recovery but waiting for the first client to connect Oct 26 18:39:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[66-67]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 Oct 26 18:39:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99 Oct 26 18:39:08 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Oct 26 18:39:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Will be in recovery for at least 2:30, or until 1424 clients reconnect Oct 26 18:39:09 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 41987afd-4763-2b68-3e1b-aa40f57d8a24 (at 10.9.114.9@o2ib4) Oct 26 18:39:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to d48e6d6f-ffa6-2902-7b7c-2094d49bc47f (at 10.8.25.18@o2ib6) Oct 26 18:39:10 oak-md1-s2 kernel: Lustre: Skipped 45 previous similar messages Oct 26 18:39:12 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 954b522d-f243-3651-ce3f-3543f276ef89 (at 10.8.21.32@o2ib6) Oct 26 18:39:12 oak-md1-s2 kernel: Lustre: Skipped 206 previous similar messages Oct 26 18:39:16 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 93e70438-cadb-6f60-0b52-66337493cda2 (at 10.9.106.10@o2ib4) Oct 26 18:39:16 oak-md1-s2 kernel: Lustre: Skipped 506 previous similar messages Oct 26 18:39:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (980 recovered, 3 in progress, and 0 evicted) to recover in 2:26 Oct 26 18:39:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 70a4db1c-78e0-ac52-e810-01b1ef553dd8 (at 10.8.26.29@o2ib6) Oct 26 18:39:24 oak-md1-s2 kernel: Lustre: Skipped 235 previous similar messages Oct 26 18:39:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to oak-MDT0001-mdtlov_UUID (at 10.0.2.51@o2ib5) Oct 26 18:39:44 oak-md1-s2 kernel: Lustre: Skipped 439 previous similar messages Oct 26 18:39:49 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (1413 recovered, 5 in progress, and 0 evicted) to recover in 2:01 Oct 26 18:40:14 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (1417 recovered, 5 in progress, and 0 evicted) to recover in 1:36 Oct 26 18:40:23 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) Oct 26 18:40:23 oak-md1-s2 kernel: Lustre: Skipped 69 previous similar messages Oct 26 18:40:38 oak-md1-s2 kernel: LustreError: 36628:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8805e54c8900 x1613417096819120/t0(0) o601->oak-MDT0000-lwp-OST002d_UUID@10.0.2.102@o2ib5:555/0 lens 336/0 e 0 to 0 dl 1540604480 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:40:38 oak-md1-s2 kernel: LustreError: 36599:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8805e54c8300 x1613417096819168/t0(0) o601->oak-MDT0000-lwp-OST002d_UUID@10.0.2.102@o2ib5:555/0 lens 336/0 e 0 to 0 dl 1540604480 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:40:38 oak-md1-s2 kernel: LustreError: 36599:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 4 previous similar messages Oct 26 18:40:38 oak-md1-s2 kernel: LustreError: 36628:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 1 previous similar message Oct 26 18:40:39 oak-md1-s2 kernel: LustreError: 36599:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88068a3d5d00 x1613417096819264/t0(0) o601->oak-MDT0000-lwp-OST0019_UUID@10.0.2.102@o2ib5:556/0 lens 336/0 e 0 to 0 dl 1540604481 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:40:39 oak-md1-s2 kernel: LustreError: 36599:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 5 previous similar messages Oct 26 18:40:39 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (1417 recovered, 5 in progress, and 0 evicted) to recover in 1:11 Oct 26 18:40:44 oak-md1-s2 kernel: LustreError: 36596:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff880cd2b7b300 x1613417096823776/t0(0) o601->oak-MDT0000-lwp-OST002f_UUID@10.0.2.102@o2ib5:561/0 lens 336/0 e 0 to 0 dl 1540604486 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:40:44 oak-md1-s2 kernel: LustreError: 36596:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 19 previous similar messages Oct 26 18:40:49 oak-md1-s2 kernel: LustreError: 36597:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff880c74bdbc00 x1613417096823888/t0(0) o601->oak-MDT0000-lwp-OST0005_UUID@10.0.2.102@o2ib5:566/0 lens 336/0 e 0 to 0 dl 1540604491 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:40:49 oak-md1-s2 kernel: LustreError: 36597:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 6 previous similar messages Oct 26 18:40:54 oak-md1-s2 kernel: LustreError: 36631:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88200c605400 x1613280543042208/t0(0) o601->oak-MDT0000-lwp-OST0026_UUID@10.0.2.101@o2ib5:570/0 lens 336/0 e 0 to 0 dl 1540604495 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:40:54 oak-md1-s2 kernel: LustreError: 36631:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 24 previous similar messages Oct 26 18:41:04 oak-md1-s2 kernel: LustreError: 36702:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff881473f6ce00 x1613280543044144/t0(0) o601->oak-MDT0000-lwp-OST0018_UUID@10.0.2.101@o2ib5:580/0 lens 336/0 e 0 to 0 dl 1540604505 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:41:04 oak-md1-s2 kernel: LustreError: 36701:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff88200c604b00 x1613280543044192/t0(0) o601->oak-MDT0000-lwp-OST0018_UUID@10.0.2.101@o2ib5:580/0 lens 336/0 e 0 to 0 dl 1540604505 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:41:04 oak-md1-s2 kernel: LustreError: 36701:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 28 previous similar messages Oct 26 18:41:04 oak-md1-s2 kernel: LustreError: 36702:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 3 previous similar messages Oct 26 18:41:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (1417 recovered, 5 in progress, and 0 evicted) to recover in 0:46 Oct 26 18:41:29 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (1417 recovered, 5 in progress, and 0 evicted) to recover in 0:21 Oct 26 18:41:38 oak-md1-s2 kernel: LustreError: 36693:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff880f191ce300 x1613417096828816/t0(0) o601->oak-MDT0000-lwp-OST002d_UUID@10.0.2.102@o2ib5:615/0 lens 336/0 e 0 to 0 dl 1540604540 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:41:38 oak-md1-s2 kernel: LustreError: 36693:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 325 previous similar messages Oct 26 18:41:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: recovery is timed out, evict stale exports Oct 26 18:41:51 oak-md1-s2 kernel: Lustre: oak-MDT0000: disconnecting 2 stale clients Oct 26 18:41:51 oak-md1-s2 kernel: LustreError: 36392:0:(update_recovery.c:1415:distribute_txn_replay_handle()) oak-MDT0000-mdtlov: undo for /tmp/rpmbuild-lustre-sthiell-tfziBdHB/BUILD/lustre-2.10.4/lustre/ptlrpc/../../lustre/target/update_recovery.c:1048: rc = -524 Oct 26 18:41:54 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (1417 recovered, 5 in progress, and 2 evicted) to recover in 1:37 Oct 26 18:42:43 oak-md1-s2 kernel: LustreError: 36737:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff880f191cf200 x1613417096834560/t0(0) o601->oak-MDT0000-lwp-OST0027_UUID@10.0.2.102@o2ib5:680/0 lens 336/0 e 0 to 0 dl 1540604605 ref 1 fl Interpret:/0/ffffffff rc 0/-1 Oct 26 18:42:43 oak-md1-s2 kernel: LustreError: 36737:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 681 previous similar messages Oct 26 18:42:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Denying connection for new client 98e37875-c7a6-805f-5b44-a64c71337dc4(at 10.8.20.15@o2ib6), waiting for 1424 known clients (1417 recovered, 5 in progress, and 2 evicted) to recover in 0:47 Oct 26 18:42:44 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 18:43:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: recovery is timed out, evict stale exports Oct 26 18:43:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: disconnecting 1 stale clients Oct 26 18:43:32 oak-md1-s2 kernel: LustreError: 36392:0:(update_recovery.c:1415:distribute_txn_replay_handle()) oak-MDT0000-mdtlov: undo for /tmp/rpmbuild-lustre-sthiell-tfziBdHB/BUILD/lustre-2.10.4/lustre/ptlrpc/../../lustre/target/update_recovery.c:1048: rc = -524 Oct 26 18:43:32 oak-md1-s2 kernel: LustreError: 36392:0:(update_recovery.c:1415:distribute_txn_replay_handle()) Skipped 1 previous similar message Oct 26 18:43:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Recovery over after 4:24, of 1424 clients 1420 recovered and 4 were evicted. Oct 26 18:43:34 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 18:43:34 oak-md1-s2 kernel: Lustre: Skipped 9 previous similar messages Oct 26 18:47:20 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fa81b78c-1920-c521-9056-cbc0887ba298 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e7e0c00, cur 1540604840 expire 1540604690 last 1540604613 Oct 26 19:00:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 19:00:50 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 26 19:01:31 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98e37875-c7a6-805f-5b44-a64c71337dc4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88097d0e9000, cur 1540605691 expire 1540605541 last 1540605464 Oct 26 19:05:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5186da8f-659f-e3cb-435b-fcfb6c1a1989 (at 10.9.112.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88202f391400, cur 1540605956 expire 1540605806 last 1540605729 Oct 26 19:06:04 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 23c0bea2-a100-c1b1-d5f5-d7c8641932c0 (at 10.9.112.3@o2ib4) Oct 26 19:07:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 19:10:30 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa81b78c-1920-c521-9056-cbc0887ba298 (at 10.8.15.1@o2ib6) Oct 26 19:12:56 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 19:13:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client fc194ede-4227-631a-8194-3b278c50a8e8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808e7ade400, cur 1540606417 expire 1540606267 last 1540606190 Oct 26 19:13:37 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 19:19:38 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client c1faf4bb-fbe8-8d2d-7c62-2d25d993844b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881e6b20a000, cur 1540606778 expire 1540606628 last 1540606551 Oct 26 19:21:44 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 19:25:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8c5d2b82-8521-bb6e-014b-d3fded24a5a4 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810c13d8400, cur 1540607157 expire 1540607007 last 1540606930 Oct 26 19:27:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 25ed85e4-6d37-4c40-f0e3-78e2241d7f18 (at 10.8.20.15@o2ib6) in 179 seconds. I think it's dead, and I am evicting it. exp ffff881e8db8dc00, cur 1540607233 expire 1540607083 last 1540607054 Oct 26 19:37:25 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 58ad6eb7-14d1-86e1-2526-cebe2e59e29a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88097d114000, cur 1540607845 expire 1540607695 last 1540607618 Oct 26 19:39:33 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 19:39:33 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 19:51:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48185c8d-3fd8-68c2-5e18-f23b26da8ae2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88153d3c6000, cur 1540608701 expire 1540608551 last 1540608474 Oct 26 19:51:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 19:51:50 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 19:55:37 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 3ad8e463-52ed-9532-2fe2-a167d0d70499 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88175d0c4800, cur 1540608937 expire 1540608787 last 1540608710 Oct 26 20:06:01 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 5846b437-c9d1-bd5f-bf55-2feaff1e27ef (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88097d1ec800, cur 1540609561 expire 1540609411 last 1540609334 Oct 26 20:11:10 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7159018a-0984-b28f-4396-c1219b5df56c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8819f3aac000, cur 1540609870 expire 1540609720 last 1540609643 Oct 26 20:11:52 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to (at 10.8.20.15@o2ib6) Oct 26 20:11:52 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 20:16:32 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 16f3ac41-6156-71b2-16cd-3efdeba0befe (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88200e7e1000, cur 1540610192 expire 1540610042 last 1540609965 Oct 26 20:19:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 359ad16e-9408-e726-2365-7b5eefb26b1b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881dda80c800, cur 1540610368 expire 1540610218 last 1540610141 Oct 26 20:24:40 oak-md1-s2 kernel: Lustre: 37112:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:24:42 oak-md1-s2 kernel: Lustre: 37183:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:24:43 oak-md1-s2 kernel: Lustre: 37112:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:24:45 oak-md1-s2 kernel: Lustre: 37181:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:24:45 oak-md1-s2 kernel: Lustre: 37181:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1 previous similar message Oct 26 20:24:50 oak-md1-s2 kernel: Lustre: 37172:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:24:50 oak-md1-s2 kernel: Lustre: 37172:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 3 previous similar messages Oct 26 20:24:59 oak-md1-s2 kernel: Lustre: 37106:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:24:59 oak-md1-s2 kernel: Lustre: 37106:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 5 previous similar messages Oct 26 20:25:15 oak-md1-s2 kernel: Lustre: 37218:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:25:15 oak-md1-s2 kernel: Lustre: 37218:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 15 previous similar messages Oct 26 20:25:49 oak-md1-s2 kernel: Lustre: 37268:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:25:49 oak-md1-s2 kernel: Lustre: 37268:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 26 previous similar messages Oct 26 20:25:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to fa81b78c-1920-c521-9056-cbc0887ba298 (at 10.8.15.1@o2ib6) Oct 26 20:25:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 20:26:54 oak-md1-s2 kernel: Lustre: 37176:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:26:54 oak-md1-s2 kernel: Lustre: 37176:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 49 previous similar messages Oct 26 20:29:03 oak-md1-s2 kernel: Lustre: 36518:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:29:03 oak-md1-s2 kernel: Lustre: 36518:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 109 previous similar messages Oct 26 20:33:19 oak-md1-s2 kernel: Lustre: 37182:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:33:19 oak-md1-s2 kernel: Lustre: 37182:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 224 previous similar messages Oct 26 20:37:19 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 16f3ac41-6156-71b2-16cd-3efdeba0befe (at 10.8.20.19@o2ib6) Oct 26 20:41:06 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client a4323a28-c323-dafd-542a-0c2cfd876ef9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880b51f93000, cur 1540611666 expire 1540611516 last 1540611439 Oct 26 20:41:52 oak-md1-s2 kernel: Lustre: 37143:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:41:52 oak-md1-s2 kernel: Lustre: 37143:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 466 previous similar messages Oct 26 20:43:41 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 98823b4f-496f-25da-7c17-e6a59d682db7 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881daf27d000, cur 1540611821 expire 1540611671 last 1540611594 Oct 26 20:44:57 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 8bdc5a3b-917d-f076-f1a2-e952000d0dec (at 10.8.20.15@o2ib6) in 210 seconds. I think it's dead, and I am evicting it. exp ffff88091d5fb000, cur 1540611897 expire 1540611747 last 1540611687 Oct 26 20:46:13 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client f243e48a-4667-0d5d-d5f8-eb4317aa4eca (at 10.8.20.19@o2ib6) in 190 seconds. I think it's dead, and I am evicting it. exp ffff880aa55be000, cur 1540611973 expire 1540611823 last 1540611783 Oct 26 20:49:03 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 16f3ac41-6156-71b2-16cd-3efdeba0befe (at 10.8.20.19@o2ib6) Oct 26 20:49:03 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 26 20:49:46 oak-md1-s2 kernel: Lustre: 37277:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 20:49:50 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7699f205-f51b-d343-be00-b8d1a4a0374b (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881246ae7800, cur 1540612190 expire 1540612040 last 1540611963 Oct 26 20:50:17 oak-md1-s2 kernel: Lustre: 37172:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 20:50:17 oak-md1-s2 kernel: Lustre: 37172:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages Oct 26 20:51:52 oak-md1-s2 kernel: Lustre: 37167:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 20:51:52 oak-md1-s2 kernel: Lustre: 37167:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 805 previous similar messages Oct 26 20:51:57 oak-md1-s2 kernel: Lustre: 37239:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 20:53:24 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 50a840a0-ea16-a865-74b8-9fb7c9366fa0 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881d9d321c00, cur 1540612404 expire 1540612254 last 1540612177 Oct 26 20:53:24 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 20:56:27 oak-md1-s2 kernel: Lustre: 37086:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 20:56:27 oak-md1-s2 kernel: Lustre: 37086:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 26 20:56:53 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 48ca92c7-8059-99ee-7429-8683f5073f12 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880ed7f19800, cur 1540612613 expire 1540612463 last 1540612386 Oct 26 20:56:53 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 20:57:10 oak-md1-s2 kernel: Lustre: 37095:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 20:57:10 oak-md1-s2 kernel: Lustre: 37095:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 40 previous similar messages Oct 26 20:57:55 oak-md1-s2 kernel: Lustre: 37145:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 20:59:18 oak-md1-s2 kernel: Lustre: 37102:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 20:59:18 oak-md1-s2 kernel: Lustre: 37102:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 37 previous similar messages Oct 26 21:01:06 oak-md1-s2 kernel: Lustre: 37118:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 21:01:06 oak-md1-s2 kernel: Lustre: 37118:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages Oct 26 21:01:53 oak-md1-s2 kernel: Lustre: 37099:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 21:01:53 oak-md1-s2 kernel: Lustre: 37099:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1172 previous similar messages Oct 26 21:02:26 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 59a0c4b8-556e-b7a8-2b7b-bb88c4227aa9 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880290ea9000, cur 1540612946 expire 1540612796 last 1540612719 Oct 26 21:02:35 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 16f3ac41-6156-71b2-16cd-3efdeba0befe (at 10.8.20.19@o2ib6) Oct 26 21:02:35 oak-md1-s2 kernel: Lustre: Skipped 4 previous similar messages Oct 26 21:05:37 oak-md1-s2 kernel: Lustre: 37192:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 21:06:22 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client aff155e2-1019-c67f-c7d5-94a5f6e9e095 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881b9489e000, cur 1540613182 expire 1540613032 last 1540612955 Oct 26 21:11:53 oak-md1-s2 kernel: Lustre: 37176:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 21:11:53 oak-md1-s2 kernel: Lustre: 37176:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 892 previous similar messages Oct 26 21:14:28 oak-md1-s2 kernel: Lustre: 37076:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 21:14:28 oak-md1-s2 kernel: Lustre: 37076:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages Oct 26 21:17:28 oak-md1-s2 kernel: Lustre: oak-MDT0000: haven't heard from client 7be3ecb9-fa37-ade5-40cd-1989084ad2b3 (at 10.8.20.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103e27d000, cur 1540613848 expire 1540613698 last 1540613621 Oct 26 21:17:28 oak-md1-s2 kernel: Lustre: Skipped 1 previous similar message Oct 26 21:17:42 oak-md1-s2 kernel: Lustre: oak-MDT0000: Connection restored to 16f3ac41-6156-71b2-16cd-3efdeba0befe (at 10.8.20.19@o2ib6) Oct 26 21:17:42 oak-md1-s2 kernel: Lustre: Skipped 2 previous similar messages Oct 26 21:19:50 oak-md1-s2 kernel: Lustre: 37106:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 Oct 26 21:19:50 oak-md1-s2 kernel: Lustre: 37106:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages Oct 26 21:21:53 oak-md1-s2 kernel: Lustre: 37167:0:(llog_cat.c:901:llog_cat_process_or_fork()) oak-MDD0000: catlog [0x7:0xa:0x0] crosses index zero Oct 26 21:21:53 oak-md1-s2 kernel: Lustre: 37167:0:(llog_cat.c:901:llog_cat_process_or_fork()) Skipped 1068 previous similar messages