[ 0.000000] microcode: microcode updated early to revision 0x43, date = 2019-03-01 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-957.27.2.el7_lustre.pl1.x86_64 (sthiell@oak-io4-s2) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Mon Aug 5 15:28:37 PDT 2019 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl1.x86_64 root=UUID=403fbadc-476b-46d5-9d30-5b0b6a8c8948 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000067b5bfff] usable [ 0.000000] BIOS-e820: [mem 0x0000000067b5c000-0x000000006fb63fff] reserved [ 0.000000] BIOS-e820: [mem 0x000000006fb64000-0x000000007a288fff] usable [ 0.000000] BIOS-e820: [mem 0x000000007a289000-0x000000007af0afff] reserved [ 0.000000] BIOS-e820: [mem 0x000000007af0b000-0x000000007b93afff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000007b93b000-0x000000007bab7fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000007bab8000-0x000000007bae8fff] usable [ 0.000000] BIOS-e820: [mem 0x000000007bae9000-0x000000007bafefff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000007baff000-0x000000007bafffff] usable [ 0.000000] BIOS-e820: [mem 0x000000007bb00000-0x000000008fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feda8000-0x00000000fedabfff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ff310000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000207fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.6.0 10/26/2017 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x2080000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask 3FE000000000 write-back [ 0.000000] 1 base 002000000000 mask 3FFF80000000 write-back [ 0.000000] 2 base 000080000000 mask 3FFF80000000 uncachable [ 0.000000] 3 base 0000FF000000 mask 3FFFFF000000 write-protect [ 0.000000] 4 disabled [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WP UC- UC [ 0.000000] x2apic enabled by BIOS, switching to x2apic ops [ 0.000000] e820: last_pfn = 0x7bb00 max_arch_pfn = 0x400000000 [ 0.000000] Base memory trampoline at [ffff8e49c0096000] 96000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x8ff253000, 0x8ff253fff] PGTABLE [ 0.000000] BRK [0x8ff254000, 0x8ff254fff] PGTABLE [ 0.000000] BRK [0x8ff255000, 0x8ff255fff] PGTABLE [ 0.000000] BRK [0x8ff256000, 0x8ff256fff] PGTABLE [ 0.000000] BRK [0x8ff257000, 0x8ff257fff] PGTABLE [ 0.000000] BRK [0x8ff258000, 0x8ff258fff] PGTABLE [ 0.000000] BRK [0x8ff259000, 0x8ff259fff] PGTABLE [ 0.000000] BRK [0x8ff25a000, 0x8ff25afff] PGTABLE [ 0.000000] BRK [0x8ff25b000, 0x8ff25bfff] PGTABLE [ 0.000000] RAMDISK: [mem 0x35bab000-0x36dcdfff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000fe320 00024 (v02 DELL ) [ 0.000000] ACPI: XSDT 000000007bab60e8 000C4 (v01 DELL PE_SC3 00000000 01000013) [ 0.000000] ACPI: FACP 000000007bab2000 000F4 (v04 DELL PE_SC3 00000000 DELL 00000001) [ 0.000000] ACPI: DSDT 000000007ba98000 10777 (v02 DELL PE_SC3 00000003 DELL 00000001) [ 0.000000] ACPI: FACS 000000007b8d3000 00040 [ 0.000000] ACPI: MCEJ 000000007bab5000 00130 (v01 INTEL 00000001 INTL 0100000D) [ 0.000000] ACPI: WD__ 000000007bab4000 00134 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: SLIC 000000007bab3000 00024 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: HPET 000000007bab1000 00038 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: APIC 000000007baaf000 01FFC (v03 DELL PE_SC3 00000000 DELL 00000001) [ 0.000000] ACPI: MCFG 000000007baae000 0003C (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: MSCT 000000007baad000 00090 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: SLIT 000000007baac000 0006C (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: SRAT 000000007baa9000 02330 (v03 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: SSDT 000000007b955000 1424A9 (v02 DELL PE_SC3 00004000 INTL 20121114) [ 0.000000] ACPI: SSDT 000000007b952000 0217F (v02 DELL PE_SC3 00000002 INTL 20121114) [ 0.000000] ACPI: SSDT 000000007b951000 0006E (v02 DELL PE_SC3 00000002 INTL 20121114) [ 0.000000] ACPI: PRAD 000000007b950000 00132 (v02 DELL PE_SC3 00000002 INTL 20121114) [ 0.000000] ACPI: SPCR 000000007bafe000 00050 (v01 00000000 00000000) [ 0.000000] ACPI: DMAR 000000007bafd000 00120 (v01 DELL PE_SC3 00000001 DELL 00000001) [ 0.000000] ACPI: HEST 000000007baf1000 0017C (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: BERT 000000007baf0000 00030 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: ERST 000000007baef000 00230 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: EINJ 000000007baee000 00150 (v01 DELL PE_SC3 00000002 DELL 00000001) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] Setting APIC routing to cluster x2apic. [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x10 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x12 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x14 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x16 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x18 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x1a -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x11 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x13 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x15 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x17 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x19 -> Node 1 [ 0.000000] SRAT: PXM 0 -> APIC 0x0b -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x1b -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x107fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=2 [ 0.000000] NODE_DATA(0) allocated [mem 0x107ffd9000-0x107fffffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x207ffd8000-0x207fffefff] [ 0.000000] Reserving 168MB of memory at 688MB for crashkernel (System RAM: 130850MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x207fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0009bfff] [ 0.000000] node 0: [mem 0x00100000-0x67b5bfff] [ 0.000000] node 0: [mem 0x6fb64000-0x7a288fff] [ 0.000000] node 0: [mem 0x7bab8000-0x7bae8fff] [ 0.000000] node 0: [mem 0x7baff000-0x7bafffff] [ 0.000000] node 0: [mem 0x100000000-0x107fffffff] [ 0.000000] node 1: [mem 0x1080000000-0x207fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x107fffffff] [ 0.000000] On node 0 totalpages: 16720462 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3995 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 7243 pages used for memmap [ 0.000000] DMA32 zone: 463539 pages, LIFO batch:31 [ 0.000000] Normal zone: 253952 pages used for memmap [ 0.000000] Normal zone: 16252928 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x1080000000-0x207fffffff] [ 0.000000] On node 1 totalpages: 16777216 [ 0.000000] Normal zone: 262144 pages used for memmap [ 0.000000] Normal zone: 16777216 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0x10] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0x12] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0x14] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0x16] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0x18] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x1a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0x11] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0x13] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0x15] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0x17] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0x19] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x1b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0xffffffff] uid[0xffffffff] disabled) [ 0.000000] ACPI: X2APIC_NMI (uid[0x00] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x01] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x02] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x03] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x04] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x05] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x06] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x07] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x08] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x09] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x0f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x10] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x11] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x12] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x13] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x14] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x15] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x16] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x17] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x18] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x19] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x1f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x20] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x21] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x22] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x23] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x24] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x25] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x26] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x27] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x28] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x29] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x2f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x30] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x31] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x32] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x33] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x34] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x35] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x36] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x37] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x38] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x39] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x3f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x40] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x41] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x42] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x43] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x44] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x45] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x46] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x47] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x48] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x49] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x4f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x50] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x51] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x52] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x53] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x54] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x55] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x56] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x57] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x58] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x59] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x5f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x60] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x61] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x62] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x63] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x64] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x65] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x66] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x67] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x68] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x69] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x6f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x70] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x71] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x72] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x73] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x74] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x75] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x76] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x77] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x78] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x79] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x7f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x80] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x81] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x82] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x83] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x84] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x85] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x86] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x87] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x88] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x89] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x8f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x90] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x91] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x92] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x93] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x94] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x95] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x96] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x97] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x98] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x99] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9a] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9b] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9c] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9d] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9e] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0x9f] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa0] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa1] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa2] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa3] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa4] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa5] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa6] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa7] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa8] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xa9] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xaa] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xab] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xac] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xad] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xae] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xaf] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb0] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb1] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb2] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb3] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb4] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb5] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb6] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb7] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb8] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xb9] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xba] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbb] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbc] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbd] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbe] high level lint[0x1]) [ 0.000000] ACPI: X2APIC_NMI (uid[0xbf] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x18] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x19] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x20] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x21] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x22] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x23] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x24] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x25] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x26] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x27] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x28] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x29] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x30] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x31] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x32] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x33] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x34] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x35] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x36] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x37] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x38] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x39] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x40] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x41] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x42] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x43] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x44] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x45] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x46] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x47] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x48] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x49] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x50] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x51] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x52] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x53] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x54] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x55] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x56] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x57] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x58] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x59] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x60] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x61] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x62] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x63] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x64] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x65] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x66] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x67] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x68] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x69] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x70] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x71] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x72] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x73] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x74] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x75] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x76] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x77] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x78] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x79] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x80] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x81] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x82] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x83] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x84] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x85] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x86] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x87] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x88] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x89] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x90] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x91] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x92] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x93] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x94] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x95] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x96] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x97] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x98] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x99] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9a] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9b] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9c] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9d] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9e] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x9f] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa0] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa1] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa2] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa3] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa4] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa5] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa6] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa7] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa8] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xa9] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xaa] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xab] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xac] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xad] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xae] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xaf] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb0] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb1] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb2] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb3] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb4] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb5] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb6] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb7] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb8] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xb9] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xba] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbb] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbc] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbd] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbe] high level lint[0x1]) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xbf] high level lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47 [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xfec40000] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 10, version 32, address 0xfec40000, GSI 48-71 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 384 CPUs, 360 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0009c000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0x67b5c000-0x6fb63fff] [ 0.000000] PM: Registered nosave memory: [mem 0x7a289000-0x7af0afff] [ 0.000000] PM: Registered nosave memory: [mem 0x7af0b000-0x7b93afff] [ 0.000000] PM: Registered nosave memory: [mem 0x7b93b000-0x7bab7fff] [ 0.000000] PM: Registered nosave memory: [mem 0x7bae9000-0x7bafefff] [ 0.000000] PM: Registered nosave memory: [mem 0x7bb00000-0x8fffffff] [ 0.000000] PM: Registered nosave memory: [mem 0x90000000-0xfeda7fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeda8000-0xfedabfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfedac000-0xff30ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff310000-0xffffffff] [ 0.000000] e820: [mem 0x90000000-0xfeda7fff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:384 nr_cpu_ids:384 nr_node_ids:2 [ 0.000000] PERCPU: Embedded 38 pages/cpu @ffff8e59fce00000 s118784 r8192 d28672 u262144 [ 0.000000] pcpu-alloc: s118784 r8192 d28672 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 000 002 004 006 008 010 012 014 [ 0.000000] pcpu-alloc: [0] 016 018 020 022 024 026 028 030 [ 0.000000] pcpu-alloc: [0] 032 034 036 038 040 042 044 046 [ 0.000000] pcpu-alloc: [0] 048 050 052 054 056 058 060 062 [ 0.000000] pcpu-alloc: [0] 064 066 068 070 072 074 076 078 [ 0.000000] pcpu-alloc: [0] 080 082 084 086 088 090 092 094 [ 0.000000] pcpu-alloc: [0] 096 098 100 102 104 106 108 110 [ 0.000000] pcpu-alloc: [0] 112 114 116 118 120 122 124 126 [ 0.000000] pcpu-alloc: [0] 128 130 132 134 136 138 140 142 [ 0.000000] pcpu-alloc: [0] 144 146 148 150 152 154 156 158 [ 0.000000] pcpu-alloc: [0] 160 162 164 166 168 170 172 174 [ 0.000000] pcpu-alloc: [0] 176 178 180 182 184 186 188 190 [ 0.000000] pcpu-alloc: [0] 192 194 196 198 200 202 204 206 [ 0.000000] pcpu-alloc: [0] 208 210 212 214 216 218 220 222 [ 0.000000] pcpu-alloc: [0] 224 226 228 230 232 234 236 238 [ 0.000000] pcpu-alloc: [0] 240 242 244 246 248 250 252 254 [ 0.000000] pcpu-alloc: [0] 256 258 260 262 264 266 268 270 [ 0.000000] pcpu-alloc: [0] 272 274 276 278 280 282 284 286 [ 0.000000] pcpu-alloc: [0] 288 290 292 294 296 298 300 302 [ 0.000000] pcpu-alloc: [0] 304 306 308 310 312 314 316 318 [ 0.000000] pcpu-alloc: [0] 320 322 324 326 328 330 332 334 [ 0.000000] pcpu-alloc: [0] 336 338 340 342 344 346 348 350 [ 0.000000] pcpu-alloc: [0] 352 354 356 358 360 362 364 366 [ 0.000000] pcpu-alloc: [0] 368 370 372 374 376 378 380 382 [ 0.000000] pcpu-alloc: [1] 001 003 005 007 009 011 013 015 [ 0.000000] pcpu-alloc: [1] 017 019 021 023 025 027 029 031 [ 0.000000] pcpu-alloc: [1] 033 035 037 039 041 043 045 047 [ 0.000000] pcpu-alloc: [1] 049 051 053 055 057 059 061 063 [ 0.000000] pcpu-alloc: [1] 065 067 069 071 073 075 077 079 [ 0.000000] pcpu-alloc: [1] 081 083 085 087 089 091 093 095 [ 0.000000] pcpu-alloc: [1] 097 099 101 103 105 107 109 111 [ 0.000000] pcpu-alloc: [1] 113 115 117 119 121 123 125 127 [ 0.000000] pcpu-alloc: [1] 129 131 133 135 137 139 141 143 [ 0.000000] pcpu-alloc: [1] 145 147 149 151 153 155 157 159 [ 0.000000] pcpu-alloc: [1] 161 163 165 167 169 171 173 175 [ 0.000000] pcpu-alloc: [1] 177 179 181 183 185 187 189 191 [ 0.000000] pcpu-alloc: [1] 193 195 197 199 201 203 205 207 [ 0.000000] pcpu-alloc: [1] 209 211 213 215 217 219 221 223 [ 0.000000] pcpu-alloc: [1] 225 227 229 231 233 235 237 239 [ 0.000000] pcpu-alloc: [1] 241 243 245 247 249 251 253 255 [ 0.000000] pcpu-alloc: [1] 257 259 261 263 265 267 269 271 [ 0.000000] pcpu-alloc: [1] 273 275 277 279 281 283 285 287 [ 0.000000] pcpu-alloc: [1] 289 291 293 295 297 299 301 303 [ 0.000000] pcpu-alloc: [1] 305 307 309 311 313 315 317 319 [ 0.000000] pcpu-alloc: [1] 321 323 325 327 329 331 333 335 [ 0.000000] pcpu-alloc: [1] 337 339 341 343 345 347 349 351 [ 0.000000] pcpu-alloc: [1] 353 355 357 359 361 363 365 367 [ 0.000000] pcpu-alloc: [1] 369 371 373 375 377 379 381 383 [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 32974254 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-957.27.2.el7_lustre.pl1.x86_64 root=UUID=403fbadc-476b-46d5-9d30-5b0b6a8c8948 ro crashkernel=auto nomodeset console=ttyS0,115200 LANG=en_US.UTF-8 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 5808184k/136314880k available (7676k kernel code, 2324168k absent, 2431812k reserved, 6045k data, 1876k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=384, Nodes=2 [ 0.000000] x86/pti: Unmapping kernel while in userspace [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=384. [ 0.000000] NR_IRQS:327936 nr_irqs:4312 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 536870912 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] spurious 8259A interrupt: IRQ7. [ 0.000000] tsc: Detected 3399.956 MHz processor [ 0.000112] Calibrating delay loop (skipped), value calculated using timer frequency.. 6799.91 BogoMIPS (lpj=3399956) [ 0.011987] pid_max: default: 393216 minimum: 3072 [ 0.018198] Security Framework initialized [ 0.022783] SELinux: Initializing. [ 0.026953] SELinux: Starting in permissive mode [ 0.026954] Yama: becoming mindful. [ 0.037167] Dentry cache hash table entries: 16777216 (order: 15, 134217728 bytes) [ 0.067461] Inode-cache hash table entries: 8388608 (order: 14, 67108864 bytes) [ 0.085132] Mount-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.093112] Mountpoint-cache hash table entries: 262144 (order: 9, 2097152 bytes) [ 0.103789] Initializing cgroup subsys memory [ 0.108725] Initializing cgroup subsys devices [ 0.113694] Initializing cgroup subsys freezer [ 0.118675] Initializing cgroup subsys net_cls [ 0.123644] Initializing cgroup subsys blkio [ 0.128418] Initializing cgroup subsys perf_event [ 0.133721] Initializing cgroup subsys hugetlb [ 0.138680] Initializing cgroup subsys pids [ 0.143358] Initializing cgroup subsys net_prio [ 0.148678] ENERGY_PERF_BIAS: Set to 'normal', was 'performance' [ 0.155391] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8) [ 0.163985] CPU0: Thermal monitoring enabled (TM1) [ 0.169377] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.175314] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0 [ 0.181348] tlb_flushall_shift: 6 [ 0.185105] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp [ 0.195808] FEATURE SPEC_CTRL Present [ 0.199903] FEATURE IBPB_SUPPORT Present [ 0.204289] Spectre V2 : Enabling Indirect Branch Prediction Barrier [ 0.211459] Spectre V2 : Mitigation: Full retpoline [ 0.216925] MDS: Mitigation: Clear CPU buffers [ 0.222645] Freeing SMP alternatives: 28k freed [ 0.229034] ACPI: Core revision 20130517 [ 0.271901] ACPI: All ACPI Tables successfully acquired [ 0.279067] ftrace: allocating 29215 entries in 115 pages [ 0.303152] DMAR: Host address width 46 [ 0.307443] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.313384] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.322231] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 0.328163] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020df [ 0.337011] DMAR: RMRR base: 0x00000067b5c000 end: 0x0000006fb63fff [ 0.344016] DMAR: RMRR base: 0x0000007ae07000 end: 0x0000007af06fff [ 0.351011] DMAR: ATSR flags: 0x0 [ 0.354718] DMAR: ATSR flags: 0x0 [ 0.358427] DMAR-IR: IOAPIC id 10 under DRHD base 0xfbffc000 IOMMU 0 [ 0.365627] DMAR-IR: IOAPIC id 8 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.372729] DMAR-IR: IOAPIC id 9 under DRHD base 0xc7ffc000 IOMMU 1 [ 0.379830] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 [ 0.385865] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [ 0.395675] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode [ 0.404507] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode [ 0.413278] DMAR-IR: Enabled IRQ remapping in x2apic mode [ 0.420104] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.436805] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz (fam: 06, model: 3f, stepping: 02) [ 0.447320] TSC deadline timer enabled [ 0.447364] Performance Events: PEBS fmt2+, Haswell events, 16-deep LBR, full-width counters, Intel PMU driver. [ 0.458682] ... version: 3 [ 0.463151] ... bit width: 48 [ 0.467719] ... generic registers: 4 [ 0.472189] ... value mask: 0000ffffffffffff [ 0.478113] ... max period: 00007fffffffffff [ 0.484037] ... fixed-purpose events: 3 [ 0.488507] ... event mask: 000000070000000f [ 0.594185] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.499878] smpboot: Booting Node 1, Processors #1 OK [ 0.603747] smpboot: Booting Node 0, Processors #2 OK [ 0.613299] smpboot: Booting Node 1, Processors #3 OK [ 0.622508] smpboot: Booting Node 0, Processors #4 OK [ 0.631742] smpboot: Booting Node 1, Processors #5 OK [ 0.640952] smpboot: Booting Node 0, Processors #6 OK [ 0.650179] smpboot: Booting Node 1, Processors #7 OK [ 0.659383] smpboot: Booting Node 0, Processors #8 OK [ 0.668629] smpboot: Booting Node 1, Processors #9 OK [ 0.677832] smpboot: Booting Node 0, Processors #10 OK [ 0.687165] smpboot: Booting Node 1, Processors #11 OK [ 0.696469] smpboot: Booting Node 0, Processors #12 [ 0.703693] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details. [ 0.719340] OK [ 0.721290] smpboot: Booting Node 1, Processors #13 OK [ 0.728889] smpboot: Booting Node 0, Processors #14 OK [ 0.736521] smpboot: Booting Node 1, Processors #15 OK [ 0.744118] smpboot: Booting Node 0, Processors #16 OK [ 0.751749] smpboot: Booting Node 1, Processors #17 OK [ 0.759350] smpboot: Booting Node 0, Processors #18 OK [ 0.766978] smpboot: Booting Node 1, Processors #19 OK [ 0.774578] smpboot: Booting Node 0, Processors #20 OK [ 0.782207] smpboot: Booting Node 1, Processors #21 OK [ 0.789807] smpboot: Booting Node 0, Processors #22 OK [ 0.797440] smpboot: Booting Node 1, Processors #23 [ 0.804442] Brought up 24 CPUs [ 0.808053] smpboot: Max logical packages: 32 [ 0.812915] smpboot: Total of 24 processors activated (163255.05 BogoMIPS) [ 1.053116] node 0 initialised, 15454373 pages in 198ms [ 1.059905] node 1 initialised, 15983302 pages in 198ms [ 1.066677] devtmpfs: initialized [ 1.079232] EVM: security.selinux [ 1.082929] EVM: security.ima [ 1.086236] EVM: security.capability [ 1.090402] PM: Registering ACPI NVS region [mem 0x7af0b000-0x7b93afff] (10682368 bytes) [ 1.101044] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 1.108716] pinctrl core: initialized pinctrl subsystem [ 1.114774] RTC time: 13:37:33, date: 08/30/19 [ 1.120137] NET: Registered protocol family 16 [ 1.125320] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 1.133768] ACPI: bus type PCI registered [ 1.138240] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 1.146003] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000) [ 1.156391] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820 [ 1.163965] pmd_set_huge: Cannot satisfy [mem 0x80000000-0x80200000] with a huge-page mapping due to MTRR override. [ 1.175948] PCI: Using configuration type 1 for base access [ 1.182172] PCI: Dell System detected, enabling pci=bfsort. [ 1.188403] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 1.205480] ACPI: Added _OSI(Module Device) [ 1.210146] ACPI: Added _OSI(Processor Device) [ 1.215101] ACPI: Added _OSI(3.0 _SCP Extensions) [ 1.220347] ACPI: Added _OSI(Processor Aggregator Device) [ 1.226369] ACPI: Added _OSI(Linux-Dell-Video) [ 1.237310] ACPI: EC: Look up EC in DSDT [ 1.355071] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [ 1.364565] ACPI: Interpreter enabled [ 1.368656] ACPI: (supports S0 S5) [ 1.372449] ACPI: Using IOAPIC for interrupt routing [ 1.378025] HEST: Table parsing has been initialized. [ 1.383661] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.393871] ACPI: Enabled 2 GPEs in block 00 to 3F [ 1.408000] ACPI: PCI Root Bridge [UNC1] (domain 0000 [bus ff]) [ 1.414608] acpi PNP0A03:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.423786] acpi PNP0A03:02: PCIe AER handled by firmware [ 1.429902] acpi PNP0A03:02: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability] [ 1.439610] acpi PNP0A03:02: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.448763] PCI host bridge to bus 0000:ff [ 1.453331] pci_bus 0000:ff: root bus resource [bus ff] [ 1.459166] pci 0000:ff:08.0: [8086:2f80] type 00 class 0x088000 [ 1.459207] pci 0000:ff:08.2: [8086:2f32] type 00 class 0x110100 [ 1.459243] pci 0000:ff:08.3: [8086:2f83] type 00 class 0x088000 [ 1.459279] pci 0000:ff:09.0: [8086:2f90] type 00 class 0x088000 [ 1.459314] pci 0000:ff:09.2: [8086:2f33] type 00 class 0x110100 [ 1.459346] pci 0000:ff:09.3: [8086:2f93] type 00 class 0x088000 [ 1.459384] pci 0000:ff:0b.0: [8086:2f81] type 00 class 0x088000 [ 1.459416] pci 0000:ff:0b.1: [8086:2f36] type 00 class 0x110100 [ 1.459447] pci 0000:ff:0b.2: [8086:2f37] type 00 class 0x110100 [ 1.459479] pci 0000:ff:0c.0: [8086:2fe0] type 00 class 0x088000 [ 1.459510] pci 0000:ff:0c.1: [8086:2fe1] type 00 class 0x088000 [ 1.459540] pci 0000:ff:0c.2: [8086:2fe2] type 00 class 0x088000 [ 1.459571] pci 0000:ff:0c.3: [8086:2fe3] type 00 class 0x088000 [ 1.459601] pci 0000:ff:0c.4: [8086:2fe4] type 00 class 0x088000 [ 1.459633] pci 0000:ff:0c.5: [8086:2fe5] type 00 class 0x088000 [ 1.459664] pci 0000:ff:0c.6: [8086:2fe6] type 00 class 0x088000 [ 1.459695] pci 0000:ff:0c.7: [8086:2fe7] type 00 class 0x088000 [ 1.459726] pci 0000:ff:0f.0: [8086:2ff8] type 00 class 0x088000 [ 1.459757] pci 0000:ff:0f.1: [8086:2ff9] type 00 class 0x088000 [ 1.459788] pci 0000:ff:0f.4: [8086:2ffc] type 00 class 0x088000 [ 1.459819] pci 0000:ff:0f.5: [8086:2ffd] type 00 class 0x088000 [ 1.459850] pci 0000:ff:0f.6: [8086:2ffe] type 00 class 0x088000 [ 1.459882] pci 0000:ff:10.0: [8086:2f1d] type 00 class 0x088000 [ 1.459913] pci 0000:ff:10.1: [8086:2f34] type 00 class 0x110100 [ 1.459946] pci 0000:ff:10.5: [8086:2f1e] type 00 class 0x088000 [ 1.459976] pci 0000:ff:10.6: [8086:2f7d] type 00 class 0x110100 [ 1.460006] pci 0000:ff:10.7: [8086:2f1f] type 00 class 0x088000 [ 1.460037] pci 0000:ff:12.0: [8086:2fa0] type 00 class 0x088000 [ 1.460070] pci 0000:ff:12.1: [8086:2f30] type 00 class 0x110100 [ 1.460102] pci 0000:ff:12.2: [8086:2f70] type 00 class 0x088000 [ 1.460137] pci 0000:ff:13.0: [8086:2fa8] type 00 class 0x088000 [ 1.460174] pci 0000:ff:13.1: [8086:2f71] type 00 class 0x088000 [ 1.460210] pci 0000:ff:13.2: [8086:2faa] type 00 class 0x088000 [ 1.460245] pci 0000:ff:13.3: [8086:2fab] type 00 class 0x088000 [ 1.460280] pci 0000:ff:13.4: [8086:2fac] type 00 class 0x088000 [ 1.460315] pci 0000:ff:13.5: [8086:2fad] type 00 class 0x088000 [ 1.460350] pci 0000:ff:13.6: [8086:2fae] type 00 class 0x088000 [ 1.460382] pci 0000:ff:13.7: [8086:2faf] type 00 class 0x088000 [ 1.460416] pci 0000:ff:14.0: [8086:2fb0] type 00 class 0x088000 [ 1.460452] pci 0000:ff:14.1: [8086:2fb1] type 00 class 0x088000 [ 1.460488] pci 0000:ff:14.2: [8086:2fb2] type 00 class 0x088000 [ 1.460523] pci 0000:ff:14.3: [8086:2fb3] type 00 class 0x088000 [ 1.460558] pci 0000:ff:14.4: [8086:2fbc] type 00 class 0x088000 [ 1.460590] pci 0000:ff:14.5: [8086:2fbd] type 00 class 0x088000 [ 1.460625] pci 0000:ff:14.6: [8086:2fbe] type 00 class 0x088000 [ 1.460657] pci 0000:ff:14.7: [8086:2fbf] type 00 class 0x088000 [ 1.460691] pci 0000:ff:15.0: [8086:2fb4] type 00 class 0x088000 [ 1.460726] pci 0000:ff:15.1: [8086:2fb5] type 00 class 0x088000 [ 1.460762] pci 0000:ff:15.2: [8086:2fb6] type 00 class 0x088000 [ 1.460797] pci 0000:ff:15.3: [8086:2fb7] type 00 class 0x088000 [ 1.460837] pci 0000:ff:16.0: [8086:2f68] type 00 class 0x088000 [ 1.460872] pci 0000:ff:16.6: [8086:2f6e] type 00 class 0x088000 [ 1.460906] pci 0000:ff:16.7: [8086:2f6f] type 00 class 0x088000 [ 1.460939] pci 0000:ff:17.0: [8086:2fd0] type 00 class 0x088000 [ 1.460975] pci 0000:ff:17.4: [8086:2fb8] type 00 class 0x088000 [ 1.461008] pci 0000:ff:17.5: [8086:2fb9] type 00 class 0x088000 [ 1.461042] pci 0000:ff:17.6: [8086:2fba] type 00 class 0x088000 [ 1.461074] pci 0000:ff:17.7: [8086:2fbb] type 00 class 0x088000 [ 1.461112] pci 0000:ff:1e.0: [8086:2f98] type 00 class 0x088000 [ 1.461143] pci 0000:ff:1e.1: [8086:2f99] type 00 class 0x088000 [ 1.461178] pci 0000:ff:1e.2: [8086:2f9a] type 00 class 0x088000 [ 1.461210] pci 0000:ff:1e.3: [8086:2fc0] type 00 class 0x088000 [ 1.461233] pci 0000:ff:1e.4: [8086:2f9c] type 00 class 0x088000 [ 1.461268] pci 0000:ff:1f.0: [8086:2f88] type 00 class 0x088000 [ 1.461302] pci 0000:ff:1f.2: [8086:2f8a] type 00 class 0x088000 [ 1.461367] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus 7f]) [ 1.467973] acpi PNP0A03:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.477154] acpi PNP0A03:03: PCIe AER handled by firmware [ 1.483281] acpi PNP0A03:03: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability] [ 1.492989] acpi PNP0A03:03: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.502142] PCI host bridge to bus 0000:7f [ 1.506710] pci_bus 0000:7f: root bus resource [bus 7f] [ 1.512542] pci 0000:7f:08.0: [8086:2f80] type 00 class 0x088000 [ 1.512575] pci 0000:7f:08.2: [8086:2f32] type 00 class 0x110100 [ 1.512608] pci 0000:7f:08.3: [8086:2f83] type 00 class 0x088000 [ 1.512642] pci 0000:7f:09.0: [8086:2f90] type 00 class 0x088000 [ 1.512674] pci 0000:7f:09.2: [8086:2f33] type 00 class 0x110100 [ 1.512704] pci 0000:7f:09.3: [8086:2f93] type 00 class 0x088000 [ 1.512740] pci 0000:7f:0b.0: [8086:2f81] type 00 class 0x088000 [ 1.512769] pci 0000:7f:0b.1: [8086:2f36] type 00 class 0x110100 [ 1.512799] pci 0000:7f:0b.2: [8086:2f37] type 00 class 0x110100 [ 1.512829] pci 0000:7f:0c.0: [8086:2fe0] type 00 class 0x088000 [ 1.512858] pci 0000:7f:0c.1: [8086:2fe1] type 00 class 0x088000 [ 1.512886] pci 0000:7f:0c.2: [8086:2fe2] type 00 class 0x088000 [ 1.512917] pci 0000:7f:0c.3: [8086:2fe3] type 00 class 0x088000 [ 1.512946] pci 0000:7f:0c.4: [8086:2fe4] type 00 class 0x088000 [ 1.512976] pci 0000:7f:0c.5: [8086:2fe5] type 00 class 0x088000 [ 1.513004] pci 0000:7f:0c.6: [8086:2fe6] type 00 class 0x088000 [ 1.513033] pci 0000:7f:0c.7: [8086:2fe7] type 00 class 0x088000 [ 1.513062] pci 0000:7f:0f.0: [8086:2ff8] type 00 class 0x088000 [ 1.513091] pci 0000:7f:0f.1: [8086:2ff9] type 00 class 0x088000 [ 1.513121] pci 0000:7f:0f.4: [8086:2ffc] type 00 class 0x088000 [ 1.513151] pci 0000:7f:0f.5: [8086:2ffd] type 00 class 0x088000 [ 1.513179] pci 0000:7f:0f.6: [8086:2ffe] type 00 class 0x088000 [ 1.513210] pci 0000:7f:10.0: [8086:2f1d] type 00 class 0x088000 [ 1.513239] pci 0000:7f:10.1: [8086:2f34] type 00 class 0x110100 [ 1.513270] pci 0000:7f:10.5: [8086:2f1e] type 00 class 0x088000 [ 1.513297] pci 0000:7f:10.6: [8086:2f7d] type 00 class 0x110100 [ 1.513326] pci 0000:7f:10.7: [8086:2f1f] type 00 class 0x088000 [ 1.513358] pci 0000:7f:12.0: [8086:2fa0] type 00 class 0x088000 [ 1.513390] pci 0000:7f:12.1: [8086:2f30] type 00 class 0x110100 [ 1.513420] pci 0000:7f:12.2: [8086:2f70] type 00 class 0x088000 [ 1.513454] pci 0000:7f:13.0: [8086:2fa8] type 00 class 0x088000 [ 1.513486] pci 0000:7f:13.1: [8086:2f71] type 00 class 0x088000 [ 1.513520] pci 0000:7f:13.2: [8086:2faa] type 00 class 0x088000 [ 1.513555] pci 0000:7f:13.3: [8086:2fab] type 00 class 0x088000 [ 1.513588] pci 0000:7f:13.4: [8086:2fac] type 00 class 0x088000 [ 1.513621] pci 0000:7f:13.5: [8086:2fad] type 00 class 0x088000 [ 1.513655] pci 0000:7f:13.6: [8086:2fae] type 00 class 0x088000 [ 1.513685] pci 0000:7f:13.7: [8086:2faf] type 00 class 0x088000 [ 1.513717] pci 0000:7f:14.0: [8086:2fb0] type 00 class 0x088000 [ 1.513750] pci 0000:7f:14.1: [8086:2fb1] type 00 class 0x088000 [ 1.513784] pci 0000:7f:14.2: [8086:2fb2] type 00 class 0x088000 [ 1.513816] pci 0000:7f:14.3: [8086:2fb3] type 00 class 0x088000 [ 1.513849] pci 0000:7f:14.4: [8086:2fbc] type 00 class 0x088000 [ 1.513880] pci 0000:7f:14.5: [8086:2fbd] type 00 class 0x088000 [ 1.513913] pci 0000:7f:14.6: [8086:2fbe] type 00 class 0x088000 [ 1.513943] pci 0000:7f:14.7: [8086:2fbf] type 00 class 0x088000 [ 1.513976] pci 0000:7f:15.0: [8086:2fb4] type 00 class 0x088000 [ 1.514009] pci 0000:7f:15.1: [8086:2fb5] type 00 class 0x088000 [ 1.514043] pci 0000:7f:15.2: [8086:2fb6] type 00 class 0x088000 [ 1.514075] pci 0000:7f:15.3: [8086:2fb7] type 00 class 0x088000 [ 1.514112] pci 0000:7f:16.0: [8086:2f68] type 00 class 0x088000 [ 1.514145] pci 0000:7f:16.6: [8086:2f6e] type 00 class 0x088000 [ 1.514177] pci 0000:7f:16.7: [8086:2f6f] type 00 class 0x088000 [ 1.514208] pci 0000:7f:17.0: [8086:2fd0] type 00 class 0x088000 [ 1.514242] pci 0000:7f:17.4: [8086:2fb8] type 00 class 0x088000 [ 1.514273] pci 0000:7f:17.5: [8086:2fb9] type 00 class 0x088000 [ 1.514306] pci 0000:7f:17.6: [8086:2fba] type 00 class 0x088000 [ 1.514337] pci 0000:7f:17.7: [8086:2fbb] type 00 class 0x088000 [ 1.514372] pci 0000:7f:1e.0: [8086:2f98] type 00 class 0x088000 [ 1.514402] pci 0000:7f:1e.1: [8086:2f99] type 00 class 0x088000 [ 1.514433] pci 0000:7f:1e.2: [8086:2f9a] type 00 class 0x088000 [ 1.514464] pci 0000:7f:1e.3: [8086:2fc0] type 00 class 0x088000 [ 1.514488] pci 0000:7f:1e.4: [8086:2f9c] type 00 class 0x088000 [ 1.514520] pci 0000:7f:1f.0: [8086:2f88] type 00 class 0x088000 [ 1.514554] pci 0000:7f:1f.2: [8086:2f8a] type 00 class 0x088000 [ 1.515353] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) [ 1.522251] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.531422] acpi PNP0A08:00: PCIe AER handled by firmware [ 1.537495] acpi PNP0A08:00: _OSC: platform does not support [SHPCHotplug] [ 1.545257] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 1.553800] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.563049] PCI host bridge to bus 0000:00 [ 1.567618] pci_bus 0000:00: root bus resource [io 0x0000-0x03bb window] [ 1.575192] pci_bus 0000:00: root bus resource [io 0x03bc-0x03df window] [ 1.582766] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7 window] [ 1.590340] pci_bus 0000:00: root bus resource [io 0x1000-0x7fff window] [ 1.597915] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.606265] pci_bus 0000:00: root bus resource [mem 0x90000000-0xc7ffbfff window] [ 1.614615] pci_bus 0000:00: root bus resource [mem 0x38000000000-0x3bfffffffff window] [ 1.623547] pci_bus 0000:00: root bus resource [bus 00-7e] [ 1.629671] pci 0000:00:00.0: [8086:2f00] type 00 class 0x060000 [ 1.629772] pci 0000:00:01.0: [8086:2f02] type 01 class 0x060400 [ 1.629833] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 1.629979] pci 0000:00:01.0: System wakeup disabled by ACPI [ 1.636319] pci 0000:00:02.0: [8086:2f04] type 01 class 0x060400 [ 1.636384] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 1.636433] pci 0000:00:02.0: System wakeup disabled by ACPI [ 1.642770] pci 0000:00:03.0: [8086:2f08] type 01 class 0x060400 [ 1.642832] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 1.642875] pci 0000:00:03.0: System wakeup disabled by ACPI [ 1.649210] pci 0000:00:03.1: [8086:2f09] type 01 class 0x060400 [ 1.649270] pci 0000:00:03.1: PME# supported from D0 D3hot D3cold [ 1.649298] pci 0000:00:03.1: System wakeup disabled by ACPI [ 1.655635] pci 0000:00:03.2: [8086:2f0a] type 01 class 0x060400 [ 1.655695] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold [ 1.655747] pci 0000:00:03.2: System wakeup disabled by ACPI [ 1.662086] pci 0000:00:05.0: [8086:2f28] type 00 class 0x088000 [ 1.662165] pci 0000:00:05.1: [8086:2f29] type 00 class 0x088000 [ 1.662255] pci 0000:00:05.2: [8086:2f2a] type 00 class 0x088000 [ 1.662331] pci 0000:00:05.4: [8086:2f2c] type 00 class 0x080020 [ 1.662337] pci 0000:00:05.4: reg 0x10: [mem 0x91f04000-0x91f04fff] [ 1.662428] pci 0000:00:11.0: [8086:8d7c] type 00 class 0xff0000 [ 1.662483] pci 0000:00:11.0: reg 0x30: [mem 0xffff0000-0xffffffff pref] [ 1.662579] pci 0000:00:11.4: [8086:8d62] type 00 class 0x010601 [ 1.662593] pci 0000:00:11.4: reg 0x10: [io 0x4078-0x407f] [ 1.662600] pci 0000:00:11.4: reg 0x14: [io 0x408c-0x408f] [ 1.662607] pci 0000:00:11.4: reg 0x18: [io 0x4070-0x4077] [ 1.662614] pci 0000:00:11.4: reg 0x1c: [io 0x4088-0x408b] [ 1.662621] pci 0000:00:11.4: reg 0x20: [io 0x4040-0x405f] [ 1.662629] pci 0000:00:11.4: reg 0x24: [mem 0x91f01000-0x91f017ff] [ 1.662682] pci 0000:00:11.4: PME# supported from D3hot [ 1.662731] pci 0000:00:16.0: [8086:8d3a] type 00 class 0x078000 [ 1.662746] pci 0000:00:16.0: reg 0x10: [mem 0x3bffff03000-0x3bffff0300f 64bit] [ 1.662816] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 1.662860] pci 0000:00:16.1: [8086:8d3b] type 00 class 0x078000 [ 1.662875] pci 0000:00:16.1: reg 0x10: [mem 0x3bffff02000-0x3bffff0200f 64bit] [ 1.662944] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 1.662999] pci 0000:00:1a.0: [8086:8d2d] type 00 class 0x0c0320 [ 1.663014] pci 0000:00:1a.0: reg 0x10: [mem 0x91f03000-0x91f033ff] [ 1.663109] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 1.663158] pci 0000:00:1c.0: [8086:8d10] type 01 class 0x060400 [ 1.663239] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 1.663288] pci 0000:00:1c.7: [8086:8d1e] type 01 class 0x060400 [ 1.663373] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold [ 1.663402] pci 0000:00:1c.7: System wakeup disabled by ACPI [ 1.669742] pci 0000:00:1d.0: [8086:8d26] type 00 class 0x0c0320 [ 1.669758] pci 0000:00:1d.0: reg 0x10: [mem 0x91f02000-0x91f023ff] [ 1.669851] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 1.669901] pci 0000:00:1f.0: [8086:8d44] type 00 class 0x060100 [ 1.670048] pci 0000:00:1f.2: [8086:8d02] type 00 class 0x010601 [ 1.670060] pci 0000:00:1f.2: reg 0x10: [io 0x4068-0x406f] [ 1.670066] pci 0000:00:1f.2: reg 0x14: [io 0x4084-0x4087] [ 1.670072] pci 0000:00:1f.2: reg 0x18: [io 0x4060-0x4067] [ 1.670078] pci 0000:00:1f.2: reg 0x1c: [io 0x4080-0x4083] [ 1.670084] pci 0000:00:1f.2: reg 0x20: [io 0x4020-0x403f] [ 1.670090] pci 0000:00:1f.2: reg 0x24: [mem 0x91f00000-0x91f007ff] [ 1.670138] pci 0000:00:1f.2: PME# supported from D3hot [ 1.670206] pci 0000:03:00.0: [1000:005f] type 00 class 0x010400 [ 1.670214] pci 0000:03:00.0: reg 0x10: [io 0x3000-0x30ff] [ 1.670221] pci 0000:03:00.0: reg 0x14: [mem 0x91e00000-0x91e0ffff 64bit] [ 1.670227] pci 0000:03:00.0: reg 0x1c: [mem 0x91d00000-0x91dfffff 64bit] [ 1.670235] pci 0000:03:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 1.670291] pci 0000:03:00.0: supports D1 D2 [ 1.670323] pci 0000:00:01.0: PCI bridge to [bus 03] [ 1.675862] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff] [ 1.675864] pci 0000:00:01.0: bridge window [mem 0x91d00000-0x91efffff] [ 1.675896] pci 0000:04:00.0: [1000:0097] type 00 class 0x010700 [ 1.675905] pci 0000:04:00.0: reg 0x10: [io 0x2000-0x20ff] [ 1.675911] pci 0000:04:00.0: reg 0x14: [mem 0x91c40000-0x91c4ffff 64bit] [ 1.675918] pci 0000:04:00.0: reg 0x1c: [mem 0x91c00000-0x91c3ffff 64bit] [ 1.675925] pci 0000:04:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 1.676033] pci 0000:04:00.0: supports D1 D2 [ 1.676052] pci 0000:04:00.0: reg 0x174: [mem 0x00000000-0x0000ffff 64bit] [ 1.676053] pci 0000:04:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR0 for 16 VFs) [ 1.687026] pci 0000:04:00.0: reg 0x17c: [mem 0x00000000-0x0003ffff 64bit] [ 1.687028] pci 0000:04:00.0: VF(n) BAR2 space: [mem 0x00000000-0x003fffff 64bit] (contains BAR2 for 16 VFs) [ 1.698044] pci 0000:00:02.0: PCI bridge to [bus 04] [ 1.703582] pci 0000:00:02.0: bridge window [io 0x2000-0x2fff] [ 1.703584] pci 0000:00:02.0: bridge window [mem 0x91c00000-0x91cfffff] [ 1.703659] pci 0000:02:00.0: [14e4:165f] type 00 class 0x020000 [ 1.703676] pci 0000:02:00.0: reg 0x10: [mem 0x91a30000-0x91a3ffff 64bit pref] [ 1.703686] pci 0000:02:00.0: reg 0x18: [mem 0x91a40000-0x91a4ffff 64bit pref] [ 1.703696] pci 0000:02:00.0: reg 0x20: [mem 0x91a50000-0x91a5ffff 64bit pref] [ 1.703703] pci 0000:02:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.703783] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold [ 1.703862] pci 0000:02:00.1: [14e4:165f] type 00 class 0x020000 [ 1.703879] pci 0000:02:00.1: reg 0x10: [mem 0x91a00000-0x91a0ffff 64bit pref] [ 1.703888] pci 0000:02:00.1: reg 0x18: [mem 0x91a10000-0x91a1ffff 64bit pref] [ 1.703898] pci 0000:02:00.1: reg 0x20: [mem 0x91a20000-0x91a2ffff 64bit pref] [ 1.703905] pci 0000:02:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.703983] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold [ 1.706593] pci 0000:00:03.0: PCI bridge to [bus 02] [ 1.712135] pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff 64bit pref] [ 1.712177] pci 0000:01:00.0: [14e4:165f] type 00 class 0x020000 [ 1.712194] pci 0000:01:00.0: reg 0x10: [mem 0x91b30000-0x91b3ffff 64bit pref] [ 1.712204] pci 0000:01:00.0: reg 0x18: [mem 0x91b40000-0x91b4ffff 64bit pref] [ 1.712214] pci 0000:01:00.0: reg 0x20: [mem 0x91b50000-0x91b5ffff 64bit pref] [ 1.712221] pci 0000:01:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.712302] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold [ 1.712369] pci 0000:01:00.1: [14e4:165f] type 00 class 0x020000 [ 1.712386] pci 0000:01:00.1: reg 0x10: [mem 0x91b00000-0x91b0ffff 64bit pref] [ 1.712396] pci 0000:01:00.1: reg 0x18: [mem 0x91b10000-0x91b1ffff 64bit pref] [ 1.712406] pci 0000:01:00.1: reg 0x20: [mem 0x91b20000-0x91b2ffff 64bit pref] [ 1.712412] pci 0000:01:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref] [ 1.712491] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold [ 1.714143] pci 0000:00:03.1: PCI bridge to [bus 01] [ 1.719683] pci 0000:00:03.1: bridge window [mem 0x91b00000-0x91bfffff 64bit pref] [ 1.719706] pci 0000:00:03.2: PCI bridge to [bus 05] [ 1.725274] pci 0000:00:1c.0: PCI bridge to [bus 06] [ 1.731171] pci 0000:07:00.0: [1912:001d] type 01 class 0x060400 [ 1.734552] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold [ 1.737961] pci 0000:00:1c.7: PCI bridge to [bus 07-0b] [ 1.743792] pci 0000:00:1c.7: bridge window [mem 0x91000000-0x919fffff] [ 1.743797] pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.745009] pci 0000:08:00.0: [1912:001d] type 01 class 0x060400 [ 1.748940] pci 0000:08:00.0: PME# supported from D0 D3hot D3cold [ 1.751444] pci 0000:07:00.0: PCI bridge to [bus 08-0b] [ 1.757468] pci 0000:07:00.0: bridge window [mem 0x91000000-0x919fffff] [ 1.757600] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.758909] pci 0000:09:00.0: [1912:001a] type 01 class 0x060400 [ 1.763629] pci 0000:08:00.0: PCI bridge to [bus 09-0a] [ 1.769688] pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff] [ 1.769844] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.771004] pci 0000:0a:00.0: [102b:0534] type 00 class 0x030000 [ 1.771487] pci 0000:0a:00.0: reg 0x10: [mem 0x90000000-0x90ffffff pref] [ 1.771766] pci 0000:0a:00.0: reg 0x14: [mem 0x91800000-0x91803fff] [ 1.772047] pci 0000:0a:00.0: reg 0x18: [mem 0x91000000-0x917fffff] [ 1.776492] pci 0000:09:00.0: PCI bridge to [bus 0a] [ 1.782254] pci 0000:09:00.0: bridge window [mem 0x91000000-0x918fffff] [ 1.782410] pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 1.783251] pci_bus 0000:00: on NUMA node 0 [ 1.783354] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 11 12 14 *15) [ 1.791554] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 *14 15) [ 1.799759] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) [ 1.807963] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 *6 7 9 10 11 12 14 15) [ 1.816162] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.825636] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.835111] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.844589] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. [ 1.854215] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) [ 1.861112] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.870288] acpi PNP0A08:01: PCIe AER handled by firmware [ 1.876405] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME PCIeCapability] [ 1.886111] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.895329] PCI host bridge to bus 0000:80 [ 1.899898] pci_bus 0000:80: root bus resource [io 0x8000-0xffff window] [ 1.907472] pci_bus 0000:80: root bus resource [mem 0xc8000000-0xfbffbfff window] [ 1.915820] pci_bus 0000:80: root bus resource [mem 0x3c000000000-0x3ffffffffff window] [ 1.924752] pci_bus 0000:80: root bus resource [bus 80-fe] [ 1.930876] pci 0000:80:01.0: [8086:2f02] type 01 class 0x060400 [ 1.930948] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold [ 1.931004] pci 0000:80:01.0: System wakeup disabled by ACPI [ 1.937342] pci 0000:80:03.0: [8086:2f08] type 01 class 0x060400 [ 1.937411] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold [ 1.937463] pci 0000:80:03.0: System wakeup disabled by ACPI [ 1.943803] pci 0000:80:05.0: [8086:2f28] type 00 class 0x088000 [ 1.943882] pci 0000:80:05.1: [8086:2f29] type 00 class 0x088000 [ 1.943974] pci 0000:80:05.2: [8086:2f2a] type 00 class 0x088000 [ 1.944048] pci 0000:80:05.4: [8086:2f2c] type 00 class 0x080020 [ 1.944054] pci 0000:80:05.4: reg 0x10: [mem 0xc8900000-0xc8900fff] [ 1.944160] pci 0000:80:01.0: PCI bridge to [bus 81] [ 1.949928] pci 0000:82:00.0: [15b3:1003] type 00 class 0x028000 [ 1.950297] pci 0000:82:00.0: reg 0x10: [mem 0xc8800000-0xc88fffff 64bit] [ 1.950512] pci 0000:82:00.0: reg 0x18: [mem 0xc8000000-0xc87fffff 64bit pref] [ 1.950913] pci 0000:82:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref] [ 1.952682] pci 0000:80:03.0: PCI bridge to [bus 82] [ 1.958220] pci 0000:80:03.0: bridge window [mem 0xc8800000-0xc88fffff] [ 1.958224] pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc87fffff 64bit pref] [ 1.958232] pci_bus 0000:80: on NUMA node 1 [ 1.958506] vgaarb: device added: PCI:0000:0a:00.0,decodes=io+mem,owns=io+mem,locks=none [ 1.967536] vgaarb: loaded [ 1.970552] vgaarb: bridge control possible 0000:0a:00.0 [ 1.976810] SCSI subsystem initialized [ 1.981011] ACPI: bus type USB registered [ 1.985493] usbcore: registered new interface driver usbfs [ 1.991616] usbcore: registered new interface driver hub [ 1.997767] usbcore: registered new device driver usb [ 2.003839] EDAC MC: Ver: 3.0.0 [ 2.007970] PCI: Using ACPI for IRQ routing [ 2.018089] PCI: pci_cache_line_size set to 64 bytes [ 2.018536] e820: reserve RAM buffer [mem 0x0009c000-0x0009ffff] [ 2.018537] e820: reserve RAM buffer [mem 0x67b5c000-0x67ffffff] [ 2.018538] e820: reserve RAM buffer [mem 0x7a289000-0x7bffffff] [ 2.018539] e820: reserve RAM buffer [mem 0x7bae9000-0x7bffffff] [ 2.018540] e820: reserve RAM buffer [mem 0x7bb00000-0x7bffffff] [ 2.018873] NetLabel: Initializing [ 2.022665] NetLabel: domain hash size = 128 [ 2.027523] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.033069] NetLabel: unlabeled traffic allowed by default [ 2.039528] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 2.046501] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 2.055032] amd_nb: Cannot enumerate AMD northbridges [ 2.060815] Switched to clocksource hpet [ 2.071486] pnp: PnP ACPI init [ 2.074892] ACPI: bus type PNP registered [ 2.079633] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.079698] system 00:01: [io 0x0500-0x053f] has been reserved [ 2.086306] system 00:01: [io 0x0400-0x047f] could not be reserved [ 2.093300] system 00:01: [io 0x0540-0x057f] has been reserved [ 2.099907] system 00:01: [io 0x0600-0x061f] has been reserved [ 2.106514] system 00:01: [io 0x0ca0-0x0ca5] has been reserved [ 2.113122] system 00:01: [io 0x0880-0x0883] has been reserved [ 2.119729] system 00:01: [io 0x0800-0x081f] has been reserved [ 2.126335] system 00:01: [mem 0xfeda8000-0xfedcbfff] could not be reserved [ 2.134107] system 00:01: [mem 0xff000000-0xffffffff] could not be reserved [ 2.141877] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved [ 2.149259] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved [ 2.156640] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved [ 2.164021] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved [ 2.171405] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.171544] pnp 00:02: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.171670] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.171716] system 00:04: [io 0x0ca8] has been reserved [ 2.177647] system 00:04: [io 0x0cac] has been reserved [ 2.183567] system 00:04: Plug and Play ACPI device, IDs IPI0001 PNP0c01 (active) [ 2.183620] pnp: PnP ACPI: found 5 devices [ 2.188181] ACPI: bus type PNP unregistered [ 2.199193] pci 0000:03:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.210261] pci 0000:04:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.221326] pci 0000:02:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.232393] pci 0000:02:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.243459] pci 0000:01:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.254524] pci 0000:01:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window [ 2.265592] pci 0000:00:11.0: can't claim BAR 6 [mem 0xffff0000-0xffffffff pref]: no compatible bridge window [ 2.276658] pci 0000:82:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window [ 2.288494] pci 0000:00:03.0: BAR 14: assigned [mem 0x92000000-0x920fffff] [ 2.296168] pci 0000:00:03.1: BAR 14: assigned [mem 0x92100000-0x921fffff] [ 2.303842] pci 0000:00:11.0: BAR 6: assigned [mem 0x91f10000-0x91f1ffff pref] [ 2.311904] pci 0000:03:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 2.319772] pci 0000:03:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 2.328027] pci 0000:00:01.0: PCI bridge to [bus 03] [ 2.333569] pci 0000:00:01.0: bridge window [io 0x3000-0x3fff] [ 2.340371] pci 0000:00:01.0: bridge window [mem 0x91d00000-0x91efffff] [ 2.347951] pci 0000:04:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 2.355819] pci 0000:04:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 2.364073] pci 0000:04:00.0: BAR 9: no space for [mem size 0x00400000 64bit] [ 2.372035] pci 0000:04:00.0: BAR 9: failed to assign [mem size 0x00400000 64bit] [ 2.380386] pci 0000:04:00.0: BAR 7: no space for [mem size 0x00100000 64bit] [ 2.388351] pci 0000:04:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit] [ 2.396703] pci 0000:00:02.0: PCI bridge to [bus 04] [ 2.402244] pci 0000:00:02.0: bridge window [io 0x2000-0x2fff] [ 2.409046] pci 0000:00:02.0: bridge window [mem 0x91c00000-0x91cfffff] [ 2.416624] pci 0000:02:00.0: BAR 6: assigned [mem 0x92000000-0x9203ffff pref] [ 2.424676] pci 0000:02:00.1: BAR 6: assigned [mem 0x92040000-0x9207ffff pref] [ 2.432736] pci 0000:00:03.0: PCI bridge to [bus 02] [ 2.438277] pci 0000:00:03.0: bridge window [mem 0x92000000-0x920fffff] [ 2.445855] pci 0000:00:03.0: bridge window [mem 0x91a00000-0x91afffff 64bit pref] [ 2.454498] pci 0000:01:00.0: BAR 6: assigned [mem 0x92100000-0x9213ffff pref] [ 2.462559] pci 0000:01:00.1: BAR 6: assigned [mem 0x92140000-0x9217ffff pref] [ 2.470619] pci 0000:00:03.1: PCI bridge to [bus 01] [ 2.476158] pci 0000:00:03.1: bridge window [mem 0x92100000-0x921fffff] [ 2.483736] pci 0000:00:03.1: bridge window [mem 0x91b00000-0x91bfffff 64bit pref] [ 2.492381] pci 0000:00:03.2: PCI bridge to [bus 05] [ 2.497923] pci 0000:00:1c.0: PCI bridge to [bus 06] [ 2.503461] pci 0000:09:00.0: PCI bridge to [bus 0a] [ 2.509147] pci 0000:09:00.0: bridge window [mem 0x91000000-0x918fffff] [ 2.516820] pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.525662] pci 0000:08:00.0: PCI bridge to [bus 09-0a] [ 2.531642] pci 0000:08:00.0: bridge window [mem 0x91000000-0x918fffff] [ 2.539312] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.548156] pci 0000:07:00.0: PCI bridge to [bus 08-0b] [ 2.554114] pci 0000:07:00.0: bridge window [mem 0x91000000-0x919fffff] [ 2.561772] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.570637] pci 0000:00:1c.7: PCI bridge to [bus 07-0b] [ 2.576460] pci 0000:00:1c.7: bridge window [mem 0x91000000-0x919fffff] [ 2.584037] pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] [ 2.592682] pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc [ 2.602779] pci_bus 0000:00: resource 4 [io 0x0000-0x03bb window] [ 2.602780] pci_bus 0000:00: resource 5 [io 0x03bc-0x03df window] [ 2.602782] pci_bus 0000:00: resource 6 [io 0x03e0-0x0cf7 window] [ 2.602783] pci_bus 0000:00: resource 7 [io 0x1000-0x7fff window] [ 2.602784] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff window] [ 2.602786] pci_bus 0000:00: resource 9 [mem 0x90000000-0xc7ffbfff window] [ 2.602787] pci_bus 0000:00: resource 10 [mem 0x38000000000-0x3bfffffffff window] [ 2.602788] pci_bus 0000:03: resource 0 [io 0x3000-0x3fff] [ 2.602790] pci_bus 0000:03: resource 1 [mem 0x91d00000-0x91efffff] [ 2.602791] pci_bus 0000:04: resource 0 [io 0x2000-0x2fff] [ 2.602792] pci_bus 0000:04: resource 1 [mem 0x91c00000-0x91cfffff] [ 2.602793] pci_bus 0000:02: resource 1 [mem 0x92000000-0x920fffff] [ 2.602795] pci_bus 0000:02: resource 2 [mem 0x91a00000-0x91afffff 64bit pref] [ 2.602796] pci_bus 0000:01: resource 1 [mem 0x92100000-0x921fffff] [ 2.602797] pci_bus 0000:01: resource 2 [mem 0x91b00000-0x91bfffff 64bit pref] [ 2.602799] pci_bus 0000:07: resource 1 [mem 0x91000000-0x919fffff] [ 2.602800] pci_bus 0000:07: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.602802] pci_bus 0000:08: resource 1 [mem 0x91000000-0x919fffff] [ 2.602803] pci_bus 0000:08: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.602804] pci_bus 0000:09: resource 1 [mem 0x91000000-0x918fffff] [ 2.602806] pci_bus 0000:09: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.602807] pci_bus 0000:0a: resource 1 [mem 0x91000000-0x918fffff] [ 2.602808] pci_bus 0000:0a: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] [ 2.602816] pci 0000:80:01.0: PCI bridge to [bus 81] [ 2.608361] pci 0000:82:00.0: BAR 6: no space for [mem size 0x00100000 pref] [ 2.616228] pci 0000:82:00.0: BAR 6: failed to assign [mem size 0x00100000 pref] [ 2.624482] pci 0000:80:03.0: PCI bridge to [bus 82] [ 2.630024] pci 0000:80:03.0: bridge window [mem 0xc8800000-0xc88fffff] [ 2.637602] pci 0000:80:03.0: bridge window [mem 0xc8000000-0xc87fffff 64bit pref] [ 2.646247] pci_bus 0000:80: resource 4 [io 0x8000-0xffff window] [ 2.646248] pci_bus 0000:80: resource 5 [mem 0xc8000000-0xfbffbfff window] [ 2.646249] pci_bus 0000:80: resource 6 [mem 0x3c000000000-0x3ffffffffff window] [ 2.646251] pci_bus 0000:82: resource 1 [mem 0xc8800000-0xc88fffff] [ 2.646252] pci_bus 0000:82: resource 2 [mem 0xc8000000-0xc87fffff 64bit pref] [ 2.646540] NET: Registered protocol family 2 [ 2.652985] TCP established hash table entries: 524288 (order: 10, 4194304 bytes) [ 2.661934] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 2.669547] TCP: Hash tables configured (established 524288 bind 65536) [ 2.676971] TCP: reno registered [ 2.680726] UDP hash table entries: 65536 (order: 9, 2097152 bytes) [ 2.688114] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes) [ 2.696830] NET: Registered protocol family 1 [ 2.715283] pci 0000:0a:00.0: Boot video device [ 2.715481] PCI: CLS 32 bytes, default 64 [ 2.715517] Unpacking initramfs... [ 2.894095] Freeing initrd memory: 18572k freed [ 2.902662] DMAR: [Firmware Bug]: RMRR entry for device 03:00.0 is broken - applying workaround [ 2.912424] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.919612] software IO TLB [mem 0x76289000-0x7a289000] (64MB) mapped at [ffff8e4a36289000-ffff8e4a3a288fff] [ 2.930964] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer [ 2.939800] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules [ 2.946213] RAPL PMU: hw unit of domain package 2^-14 Joules [ 2.952527] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 2.968194] sha1_ssse3: Using AVX2 optimized SHA-1 implementation [ 2.975040] sha256_ssse3: Using AVX2 optimized SHA-256 implementation [ 2.985405] futex hash table entries: 131072 (order: 11, 8388608 bytes) [ 2.993777] Initialise system trusted keyring [ 2.998745] audit: initializing netlink socket (disabled) [ 3.004777] type=2000 audit(1567172247.500:1): initialized [ 3.028262] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 3.035354] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 3.043438] zpool: loaded [ 3.046360] zbud: loaded [ 3.049963] VFS: Disk quotas dquot_6.6.0 [ 3.054803] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 3.063264] msgmni has been set to 32768 [ 3.067946] Key type big_key registered [ 3.072225] SELinux: Registering netfilter hooks [ 3.073252] NET: Registered protocol family 38 [ 3.078216] Key type asymmetric registered [ 3.082788] Asymmetric key parser 'x509' registered [ 3.088330] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 248) [ 3.096772] io scheduler noop registered [ 3.101150] io scheduler deadline registered (default) [ 3.106986] io scheduler cfq registered [ 3.111265] io scheduler mq-deadline registered [ 3.116320] io scheduler kyber registered [ 3.121237] pcieport 0000:00:01.0: irq 25 for MSI/MSI-X [ 3.121416] pcieport 0000:00:02.0: irq 26 for MSI/MSI-X [ 3.121584] pcieport 0000:00:03.0: irq 27 for MSI/MSI-X [ 3.121750] pcieport 0000:00:03.1: irq 28 for MSI/MSI-X [ 3.121923] pcieport 0000:00:03.2: irq 29 for MSI/MSI-X [ 3.122191] pcieport 0000:00:1c.0: irq 30 for MSI/MSI-X [ 3.122465] pcieport 0000:00:1c.7: irq 31 for MSI/MSI-X [ 3.125375] pcieport 0000:80:01.0: irq 33 for MSI/MSI-X [ 3.125573] pcieport 0000:80:03.0: irq 34 for MSI/MSI-X [ 3.125637] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 3.133410] pci 0000:03:00.0: Signaling PME through PCIe PME interrupt [ 3.140689] pcie_pme 0000:00:01.0:pcie001: service driver pcie_pme loaded [ 3.140698] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt [ 3.148467] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt [ 3.155753] pcie_pme 0000:00:02.0:pcie001: service driver pcie_pme loaded [ 3.155763] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt [ 3.163533] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 3.170817] pci 0000:02:00.1: Signaling PME through PCIe PME interrupt [ 3.178103] pcie_pme 0000:00:03.0:pcie001: service driver pcie_pme loaded [ 3.178113] pcieport 0000:00:03.1: Signaling PME through PCIe PME interrupt [ 3.185883] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt [ 3.193166] pci 0000:01:00.1: Signaling PME through PCIe PME interrupt [ 3.200452] pcie_pme 0000:00:03.1:pcie001: service driver pcie_pme loaded [ 3.200461] pcieport 0000:00:03.2: Signaling PME through PCIe PME interrupt [ 3.208232] pcie_pme 0000:00:03.2:pcie001: service driver pcie_pme loaded [ 3.208244] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 3.216017] pcie_pme 0000:00:1c.0:pcie001: service driver pcie_pme loaded [ 3.216030] pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt [ 3.223833] pcieport 0000:07:00.0: Signaling PME through PCIe PME interrupt [ 3.231606] pcieport 0000:08:00.0: Signaling PME through PCIe PME interrupt [ 3.239370] pci 0000:09:00.0: Signaling PME through PCIe PME interrupt [ 3.246647] pci 0000:0a:00.0: Signaling PME through PCIe PME interrupt [ 3.253933] pcie_pme 0000:00:1c.7:pcie001: service driver pcie_pme loaded [ 3.264845] pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt [ 3.272618] pcie_pme 0000:80:01.0:pcie001: service driver pcie_pme loaded [ 3.272653] pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt [ 3.280423] pci 0000:82:00.0: Signaling PME through PCIe PME interrupt [ 3.287711] pcie_pme 0000:80:03.0:pcie001: service driver pcie_pme loaded [ 3.287757] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 3.293982] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 3.301614] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.309147] intel_idle: MWAIT substates: 0x2120 [ 3.309148] intel_idle: v0.4.1 model 0x3F [ 3.309357] intel_idle: lapic_timer_reliable_states 0xffffffff [ 3.309450] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.317708] ACPI: Power Button [PWRF] [ 3.322315] ERST: Error Record Serialization Table (ERST) support is initialized. [ 3.330670] pstore: Registered erst as persistent store backend [ 3.337537] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 3.345850] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.373370] 00:02: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 3.400129] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 3.406658] Non-volatile memory driver v1.3 [ 3.411347] Linux agpgart interface v0.103 [ 3.416070] crash memory driver: version 1.1 [ 3.420950] rdac: device handler registered [ 3.425645] hp_sw: device handler registered [ 3.430403] emc: device handler registered [ 3.435164] alua: device handler registered [ 3.439858] libphy: Fixed MDIO Bus: probed [ 3.444459] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 3.451748] ehci-pci: EHCI PCI platform driver [ 3.456887] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 3.462752] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 3.471018] ehci-pci 0000:00:1a.0: debug port 2 [ 3.479975] ehci-pci 0000:00:1a.0: cache line size of 32 is not supported [ 3.479984] ehci-pci 0000:00:1a.0: irq 18, io mem 0x91f03000 [ 3.491793] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 3.498229] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 3.505807] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 3.513866] usb usb1: Product: EHCI Host Controller [ 3.519310] usb usb1: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl1.x86_64 ehci_hcd [ 3.528341] usb usb1: SerialNumber: 0000:00:1a.0 [ 3.533567] hub 1-0:1.0: USB hub found [ 3.537754] hub 1-0:1.0: 2 ports detected [ 3.542374] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 3.548238] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 3.556502] ehci-pci 0000:00:1d.0: debug port 2 [ 3.565457] ehci-pci 0000:00:1d.0: cache line size of 32 is not supported [ 3.565460] ehci-pci 0000:00:1d.0: irq 18, io mem 0x91f02000 [ 3.576794] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 3.583223] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 [ 3.590800] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 3.598860] usb usb2: Product: EHCI Host Controller [ 3.604303] usb usb2: Manufacturer: Linux 3.10.0-957.27.2.el7_lustre.pl1.x86_64 ehci_hcd [ 3.613335] usb usb2: SerialNumber: 0000:00:1d.0 [ 3.618549] hub 2-0:1.0: USB hub found [ 3.622736] hub 2-0:1.0: 2 ports detected [ 3.627297] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 3.634200] ohci-pci: OHCI PCI platform driver [ 3.639173] uhci_hcd: USB Universal Host Controller Interface driver [ 3.646325] usbcore: registered new interface driver usbserial_generic [ 3.653617] usbserial: USB Serial support registered for generic [ 3.660343] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 3.967805] tsc: Refined TSC clocksource calibration: 3399.996 MHz [ 4.701535] i8042: No controller found [ 4.705792] Switched to clocksource tsc [ 4.705803] mousedev: PS/2 mouse device common for all mice [ 4.705904] rtc_cmos 00:00: RTC can wake from S4 [ 4.706036] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0 [ 4.706067] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram, hpet irqs [ 4.706109] intel_pstate: Intel P-state driver initializing [ 4.710519] cpuidle: using governor menu [ 4.711143] hidraw: raw HID events driver (C) Jiri Kosina [ 4.711329] usbcore: registered new interface driver usbhid [ 4.711330] usbhid: USB HID core driver [ 4.711511] drop_monitor: Initializing network drop monitor service [ 4.712106] TCP: cubic registered [ 4.712117] Initializing XFRM netlink socket [ 4.712752] NET: Registered protocol family 10 [ 4.714647] NET: Registered protocol family 17 [ 4.714654] mpls_gso: MPLS GSO support [ 4.716337] intel_rdt: Intel RDT L3 monitoring detected [ 4.716342] mce: Using 22 MCE banks [ 4.716405] microcode: sig=0x306f2, pf=0x1, revision=0x43 [ 4.721510] microcode: Microcode Update Driver: v2.01 <tigran@aivazian.fsnet.co.uk>, Peter Oruba [ 4.721619] PM: Hibernation image not present or could not be loaded. [ 4.721622] Loading compiled-in X.509 certificates [ 4.721641] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 4.721655] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 4.722057] Loaded X.509 cert 'CentOS Linux kernel signing key: 9e53aba22e464fccc5bb7396174083706426f6e2' [ 4.722071] registered taskstats version 1 [ 4.724367] Key type trusted registered [ 4.725956] Key type encrypted registered [ 4.726042] IMA: No TPM chip found, activating TPM-bypass! (rc=-19) [ 4.728436] Magic number: 15:227:636 [ 4.736796] rtc_cmos 00:00: setting system clock to 2019-08-30 13:37:36 UTC (1567172256) [ 4.816817] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 4.899048] Freeing unused kernel memory: 1876k freed [ 4.905086] Write protecting the kernel read-only data: 12288k [ 4.913159] Freeing unused kernel memory: 504k freed [ 4.916849] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 4.927564] Freeing unused kernel memory: 596k freed [ 4.939787] random: systemd: uninitialized urandom read (16 bytes read) [ 4.941240] usb 1-1: New USB device found, idVendor=8087, idProduct=800a [ 4.941243] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 4.941528] hub 1-1:1.0: USB hub found [ 4.941609] hub 1-1:1.0: 6 ports detected [ 4.974936] random: systemd: uninitialized urandom read (16 bytes read) [ 4.982345] random: systemd: uninitialized urandom read (16 bytes read) [ 4.991622] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 5.012673] systemd[1]: Detected architecture x86-64. [ 5.018345] systemd[1]: Running in initial RAM disk. [ 5.032980] systemd[1]: Set hostname to <oak-md1-s2>. [ 5.041240] usb 2-1: New USB device found, idVendor=8087, idProduct=8002 [ 5.048732] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 5.057034] hub 2-1:1.0: USB hub found [ 5.061372] hub 2-1:1.0: 8 ports detected [ 5.091196] random: systemd: uninitialized urandom read (16 bytes read) [ 5.098617] random: systemd: uninitialized urandom read (16 bytes read) [ 5.106056] random: systemd: uninitialized urandom read (16 bytes read) [ 5.113526] random: systemd: uninitialized urandom read (16 bytes read) [ 5.121844] random: systemd: uninitialized urandom read (16 bytes read) [ 5.129440] random: systemd: uninitialized urandom read (16 bytes read) [ 5.137249] random: systemd: uninitialized urandom read (16 bytes read) [ 5.149897] systemd[1]: Reached target Timers. [ 5.159935] systemd[1]: Reached target Local File Systems. [ 5.172905] systemd[1]: Reached target Swap. [ 5.183271] systemd[1]: Created slice Root Slice. [ 5.194959] systemd[1]: Listening on udev Kernel Socket. [ 5.207027] systemd[1]: Listening on Journal Socket. [ 5.218963] systemd[1]: Listening on udev Control Socket. [ 5.230903] systemd[1]: Reached target Sockets. [ 5.238857] usb 1-1.6: new high-speed USB device number 3 using ehci-pci [ 5.248015] systemd[1]: Created slice System Slice. [ 5.260780] systemd[1]: Starting Load Kernel Modules... [ 5.270969] systemd[1]: Reached target Slices. [ 5.281608] systemd[1]: Starting Journal Service... [ 5.292589] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 5.313640] systemd[1]: Starting dracut cmdline hook... [ 5.324600] systemd[1]: Starting Setup Virtual Console... [ 5.336252] usb 1-1.6: New USB device found, idVendor=413c, idProduct=a001 [ 5.336255] usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 5.336258] usb 1-1.6: Product: Gadget USB HUB [ 5.336260] usb 1-1.6: Manufacturer: no manufacturer [ 5.336262] usb 1-1.6: SerialNumber: 0123456789 [ 5.336359] systemd[1]: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE [ 5.336586] systemd[1]: Failed to start Load Kernel Modules. [ 5.336683] hub 1-1.6:1.0: USB hub found [ 5.336874] hub 1-1.6:1.0: 6 ports detected [ 5.416912] systemd[1]: Unit systemd-modules-load.service entered failed state. [ 5.425283] systemd[1]: systemd-modules-load.service failed. [ 5.431911] systemd[1]: Started Create list of required static device nodes for the current kernel. [ 5.452224] systemd[1]: Started dracut cmdline hook. [ 5.463142] systemd[1]: Started Journal Service. [ 5.554145] random: fast init done [ 5.658227] pps_core: LinuxPPS API ver. 1 registered [ 5.664193] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> [ 5.678598] mlx_compat: loading out-of-tree module taints kernel. [ 5.686018] mlx_compat: module verification failed: signature and/or required key missing - tainting kernel [ 5.686102] megasas: 07.705.02.00-rh1 [ 5.686364] megaraid_sas 0000:03:00.0: FW now in Ready state [ 5.686366] megaraid_sas 0000:03:00.0: 64 bit DMA mask and 32 bit consistent mask [ 5.686763] megaraid_sas 0000:03:00.0: irq 36 for MSI/MSI-X [ 5.686824] megaraid_sas 0000:03:00.0: irq 37 for MSI/MSI-X [ 5.686853] megaraid_sas 0000:03:00.0: irq 38 for MSI/MSI-X [ 5.686882] megaraid_sas 0000:03:00.0: irq 39 for MSI/MSI-X [ 5.686911] megaraid_sas 0000:03:00.0: irq 40 for MSI/MSI-X [ 5.686977] megaraid_sas 0000:03:00.0: irq 41 for MSI/MSI-X [ 5.687007] megaraid_sas 0000:03:00.0: irq 42 for MSI/MSI-X [ 5.687036] megaraid_sas 0000:03:00.0: irq 43 for MSI/MSI-X [ 5.687065] megaraid_sas 0000:03:00.0: irq 44 for MSI/MSI-X [ 5.687095] megaraid_sas 0000:03:00.0: irq 45 for MSI/MSI-X [ 5.687124] megaraid_sas 0000:03:00.0: irq 46 for MSI/MSI-X [ 5.687153] megaraid_sas 0000:03:00.0: irq 47 for MSI/MSI-X [ 5.687185] megaraid_sas 0000:03:00.0: irq 48 for MSI/MSI-X [ 5.687218] megaraid_sas 0000:03:00.0: irq 49 for MSI/MSI-X [ 5.687250] megaraid_sas 0000:03:00.0: irq 50 for MSI/MSI-X [ 5.687281] megaraid_sas 0000:03:00.0: irq 51 for MSI/MSI-X [ 5.687312] megaraid_sas 0000:03:00.0: irq 52 for MSI/MSI-X [ 5.687343] megaraid_sas 0000:03:00.0: irq 53 for MSI/MSI-X [ 5.687374] megaraid_sas 0000:03:00.0: irq 54 for MSI/MSI-X [ 5.687405] megaraid_sas 0000:03:00.0: irq 55 for MSI/MSI-X [ 5.687436] megaraid_sas 0000:03:00.0: irq 56 for MSI/MSI-X [ 5.687506] megaraid_sas 0000:03:00.0: irq 57 for MSI/MSI-X [ 5.687538] megaraid_sas 0000:03:00.0: irq 58 for MSI/MSI-X [ 5.687570] megaraid_sas 0000:03:00.0: irq 59 for MSI/MSI-X [ 5.687652] megaraid_sas 0000:03:00.0: firmware supports msix : (96) [ 5.687653] megaraid_sas 0000:03:00.0: current msix/online cpus : (24/24) [ 5.687655] megaraid_sas 0000:03:00.0: RDPQ mode : (disabled) [ 5.687657] megaraid_sas 0000:03:00.0: Current firmware supports maximum commands: 928 LDIO threshold: 237 [ 5.687939] megaraid_sas 0000:03:00.0: Configured max firmware commands: 927 [ 5.691762] megaraid_sas 0000:03:00.0: FW supports sync cache : No [ 5.727070] PTP clock support registered [ 5.776601] Compat-mlnx-ofed backport release: b4fdfac [ 5.776790] tg3.c:v3.137 (May 11, 2014) [ 5.786681] Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git b4fdfac [ 5.788172] tg3 0000:01:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f2 [ 5.788175] tg3 0000:01:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.788177] tg3 0000:01:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.788178] tg3 0000:01:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.800378] tg3 0000:01:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f3 [ 5.800381] tg3 0000:01:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.800383] tg3 0000:01:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.800385] tg3 0000:01:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.816653] tg3 0000:02:00.0 eth2: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f4 [ 5.816656] tg3 0000:02:00.0 eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.816658] tg3 0000:02:00.0 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.816660] tg3 0000:02:00.0 eth2: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.828318] tg3 0000:02:00.1 eth3: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 14:18:77:60:d9:f5 [ 5.828320] tg3 0000:02:00.1 eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) [ 5.828321] tg3 0000:02:00.1 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] [ 5.828322] tg3 0000:02:00.1 eth3: dma_rwctrl[00000001] dma_mask[64-bit] [ 5.873559] compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git [ 5.932966] libata version 3.00 loaded. [ 5.977655] ahci 0000:00:11.4: version 3.0 [ 5.977897] ahci 0000:00:11.4: irq 64 for MSI/MSI-X [ 5.977909] ahci 0000:00:11.4: SSS flag set, parallel bus scan disabled [ 5.986023] mpt3sas version 26.00.00.00 loaded [ 5.992583] mpt3sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (131580476 kB) [ 6.003994] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode [ 6.004054] mlx4_core: Mellanox ConnectX core driver v4.5-1.0.1 [ 6.004082] mlx4_core: Initializing 0000:82:00.0 [ 6.028526] ahci 0000:00:11.4: flags: 64bit ncq stag led clo pio slum part ems apst [ 6.036127] scsi host2: ahci [ 6.036202] scsi host3: ahci [ 6.036271] scsi host4: ahci [ 6.036332] scsi host5: ahci [ 6.036357] ata1: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01100 irq 64 [ 6.036358] ata2: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01180 irq 64 [ 6.036361] ata3: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01200 irq 64 [ 6.036363] ata4: SATA max UDMA/133 abar m2048@0x91f01000 port 0x91f01280 irq 64 [ 6.036555] ahci 0000:00:1f.2: irq 67 for MSI/MSI-X [ 6.036564] ahci 0000:00:1f.2: SSS flag set, parallel bus scan disabled [ 6.045890] megaraid_sas 0000:03:00.0: Init cmd return status SUCCESS for SCSI host 0 [ 6.046901] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 6.046903] ahci 0000:00:1f.2: flags: 64bit ncq stag led clo pio slum part ems apst [ 6.057275] scsi host6: ahci [ 6.057350] scsi host7: ahci [ 6.057421] scsi host8: ahci [ 6.057480] scsi host9: ahci [ 6.057541] scsi host10: ahci [ 6.057603] scsi host11: ahci [ 6.057628] ata5: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00100 irq 67 [ 6.057630] ata6: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00180 irq 67 [ 6.057632] ata7: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00200 irq 67 [ 6.057634] ata8: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00280 irq 67 [ 6.057635] ata9: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00300 irq 67 [ 6.057636] ata10: SATA max UDMA/133 abar m2048@0x91f00000 port 0x91f00380 irq 67 [ 6.066846] megaraid_sas 0000:03:00.0: firmware type : Legacy(64 VD) firmware [ 6.066847] megaraid_sas 0000:03:00.0: controller type : iMR(0MB) [ 6.066848] megaraid_sas 0000:03:00.0: Online Controller Reset(OCR) : Enabled [ 6.066849] megaraid_sas 0000:03:00.0: Secure JBOD support : No [ 6.066849] megaraid_sas 0000:03:00.0: NVMe passthru support : No [ 6.089075] megaraid_sas 0000:03:00.0: INIT adapter done [ 6.089077] megaraid_sas 0000:03:00.0: Jbod map is not supported megasas_setup_jbod_map 5146 [ 6.108866] mpt3sas_cm0: IOC Number : 0 [ 6.108867] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 6.109176] mpt3sas 0000:04:00.0: irq 68 for MSI/MSI-X [ 6.109194] mpt3sas 0000:04:00.0: irq 69 for MSI/MSI-X [ 6.109212] mpt3sas 0000:04:00.0: irq 70 for MSI/MSI-X [ 6.109231] mpt3sas 0000:04:00.0: irq 71 for MSI/MSI-X [ 6.109250] mpt3sas 0000:04:00.0: irq 74 for MSI/MSI-X [ 6.109307] mpt3sas 0000:04:00.0: irq 75 for MSI/MSI-X [ 6.109325] mpt3sas 0000:04:00.0: irq 76 for MSI/MSI-X [ 6.109345] mpt3sas 0000:04:00.0: irq 77 for MSI/MSI-X [ 6.109364] mpt3sas 0000:04:00.0: irq 78 for MSI/MSI-X [ 6.109382] mpt3sas 0000:04:00.0: irq 79 for MSI/MSI-X [ 6.109400] mpt3sas 0000:04:00.0: irq 80 for MSI/MSI-X [ 6.109418] mpt3sas 0000:04:00.0: irq 81 for MSI/MSI-X [ 6.109437] mpt3sas 0000:04:00.0: irq 82 for MSI/MSI-X [ 6.109456] mpt3sas 0000:04:00.0: irq 83 for MSI/MSI-X [ 6.109474] mpt3sas 0000:04:00.0: irq 84 for MSI/MSI-X [ 6.109492] mpt3sas 0000:04:00.0: irq 85 for MSI/MSI-X [ 6.109511] mpt3sas 0000:04:00.0: irq 86 for MSI/MSI-X [ 6.109529] mpt3sas 0000:04:00.0: irq 87 for MSI/MSI-X [ 6.109547] mpt3sas 0000:04:00.0: irq 88 for MSI/MSI-X [ 6.109565] mpt3sas 0000:04:00.0: irq 89 for MSI/MSI-X [ 6.109584] mpt3sas 0000:04:00.0: irq 90 for MSI/MSI-X [ 6.109686] mpt3sas 0000:04:00.0: irq 91 for MSI/MSI-X [ 6.109703] mpt3sas 0000:04:00.0: irq 92 for MSI/MSI-X [ 6.109721] mpt3sas 0000:04:00.0: irq 93 for MSI/MSI-X [ 6.110102] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 68 [ 6.110103] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 69 [ 6.110103] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 70 [ 6.110103] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 71 [ 6.110104] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 74 [ 6.110104] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 75 [ 6.110105] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 76 [ 6.110105] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 77 [ 6.110106] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 78 [ 6.110106] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 79 [ 6.110106] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 80 [ 6.110107] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 81 [ 6.110107] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 82 [ 6.110108] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 83 [ 6.110108] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 84 [ 6.110108] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 85 [ 6.110109] mpt3sas0-msix16: PCI-MSI-X enabled: IRQ 86 [ 6.110109] mpt3sas0-msix17: PCI-MSI-X enabled: IRQ 87 [ 6.110109] mpt3sas0-msix18: PCI-MSI-X enabled: IRQ 88 [ 6.110110] mpt3sas0-msix19: PCI-MSI-X enabled: IRQ 89 [ 6.110110] mpt3sas0-msix20: PCI-MSI-X enabled: IRQ 90 [ 6.110111] mpt3sas0-msix21: PCI-MSI-X enabled: IRQ 91 [ 6.110111] mpt3sas0-msix22: PCI-MSI-X enabled: IRQ 92 [ 6.110111] mpt3sas0-msix23: PCI-MSI-X enabled: IRQ 93 [ 6.110112] mpt3sas_cm0: iomem(0x0000000091c40000), mapped(0xffffa6afcd7a0000), size(65536) [ 6.110113] mpt3sas_cm0: ioport(0x0000000000002000), size(256) [ 6.115787] megaraid_sas 0000:03:00.0: pci id : (0x1000)/(0x005f)/(0x1028)/(0x1f4b) [ 6.115788] megaraid_sas 0000:03:00.0: unevenspan support : yes [ 6.115789] megaraid_sas 0000:03:00.0: firmware crash dump : no [ 6.115789] megaraid_sas 0000:03:00.0: jbod sync map : no [ 6.115793] scsi host0: Avago SAS based MegaRAID driver [ 6.153125] scsi 0:2:0:0: Direct-Access DELL PERC H330 Mini 4.28 PQ: 0 ANSI: 5 [ 6.201848] mpt3sas_cm0: IOC Number : 0 [ 6.201848] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k [ 6.341887] ata1: SATA link down (SStatus 0 SControl 300) [ 6.352599] mpt3sas_cm0: Allocated physical memory: size(29776 kB) [ 6.352600] mpt3sas_cm0: Current Controller Queue Depth(9564), Max Controller Queue Depth(9664) [ 6.352601] mpt3sas_cm0: Scatter Gather Elements per IO(128) [ 6.361882] ata5: SATA link down (SStatus 0 SControl 300) [ 6.534740] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.01.00), ChipRevision(0x02), BiosVersion(04.00.00.00) [ 6.545438] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ) [ 6.558673] mpt3sas_cm0: : host protection capabilities enabled DIF1 DIF2 DIF3 [ 6.566889] scsi host1: Fusion MPT SAS Host [ 6.571761] mpt3sas_cm0: sending port enable !! [ 6.650882] ata2: SATA link down (SStatus 0 SControl 300) [ 6.961884] ata3: SATA link down (SStatus 0 SControl 300) [ 7.272888] ata4: SATA link down (SStatus 0 SControl 300) [ 7.583889] ata6: SATA link down (SStatus 0 SControl 300) [ 7.894893] ata7: SATA link down (SStatus 0 SControl 300) [ 8.205898] ata8: SATA link down (SStatus 0 SControl 300) [ 8.421528] mpt3sas_cm0: hba_port entry: ffff8e59fa680f00, port: 255 is added to hba_port list [ 8.432701] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b00aafc090), phys(8) [ 8.442274] mpt3sas_cm0: detecting: handle(0x0009), sas_address(0x500a09849de54f10), phy(0) [ 8.451601] mpt3sas_cm0: REPORT_LUNS: handle(0x0009), retries(0) [ 8.458453] mpt3sas_cm0: REPORT_LUNS: handle(0x0009), retries(1) [ 8.466095] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0009), lun(0) [ 8.473107] mpt3sas_cm0: detecting: handle(0x0009), sas_address(0x500a09849de54f10), phy(0) [ 8.482432] mpt3sas_cm0: REPORT_LUNS: handle(0x0009), retries(0) [ 8.489753] mpt3sas_cm0: TEST_UNIT_READY: handle(0x0009), lun(0) [ 8.497215] scsi 1:0:0:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 8.506327] scsi 1:0:0:0: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) [ 8.516915] ata9: SATA link down (SStatus 0 SControl 300) [ 8.524084] scsi 1:0:0:0: enclosure logical id(0x500605b00aafc090), slot(3) [ 8.531949] scsi 1:0:0:0: enclosure level(0x0000), connector name( ) [ 8.539433] scsi 1:0:0:0: serial_number(021546008063 ) [ 8.545463] scsi 1:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 8.821904] ata10: SATA link down (SStatus 0 SControl 300) [ 11.893796] mlx4_core: device is working in RoCE mode: Roce V1 [ 11.900308] mlx4_core: UD QP Gid type is: V1 [ 12.208976] scsi 1:0:0:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 12.218093] scsi 1:0:0:1: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) [ 12.229840] scsi 1:0:0:1: enclosure logical id(0x500605b00aafc090), slot(3) [ 12.237707] scsi 1:0:0:1: enclosure level(0x0000), connector name( ) [ 12.245194] scsi 1:0:0:1: serial_number(021546008063 ) [ 12.251222] scsi 1:0:0:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 13.552277] mlx4_core 0000:82:00.0: DMFS high rate steer mode is: default performance [ 13.561299] mlx4_core 0000:82:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link) [ 13.570961] mlx4_core 0000:82:00.0: irq 94 for MSI/MSI-X [ 13.570980] mlx4_core 0000:82:00.0: irq 95 for MSI/MSI-X [ 13.571000] mlx4_core 0000:82:00.0: irq 96 for MSI/MSI-X [ 13.571020] mlx4_core 0000:82:00.0: irq 97 for MSI/MSI-X [ 13.571039] mlx4_core 0000:82:00.0: irq 98 for MSI/MSI-X [ 13.571058] mlx4_core 0000:82:00.0: irq 99 for MSI/MSI-X [ 13.571077] mlx4_core 0000:82:00.0: irq 100 for MSI/MSI-X [ 13.571098] mlx4_core 0000:82:00.0: irq 101 for MSI/MSI-X [ 13.571118] mlx4_core 0000:82:00.0: irq 102 for MSI/MSI-X [ 13.571137] mlx4_core 0000:82:00.0: irq 103 for MSI/MSI-X [ 13.571156] mlx4_core 0000:82:00.0: irq 104 for MSI/MSI-X [ 13.571175] mlx4_core 0000:82:00.0: irq 105 for MSI/MSI-X [ 13.571193] mlx4_core 0000:82:00.0: irq 106 for MSI/MSI-X [ 13.571212] mlx4_core 0000:82:00.0: irq 107 for MSI/MSI-X [ 13.571231] mlx4_core 0000:82:00.0: irq 108 for MSI/MSI-X [ 13.571251] mlx4_core 0000:82:00.0: irq 109 for MSI/MSI-X [ 13.571270] mlx4_core 0000:82:00.0: irq 110 for MSI/MSI-X [ 13.571289] mlx4_core 0000:82:00.0: irq 111 for MSI/MSI-X [ 13.571308] mlx4_core 0000:82:00.0: irq 112 for MSI/MSI-X [ 13.571326] mlx4_core 0000:82:00.0: irq 113 for MSI/MSI-X [ 13.571346] mlx4_core 0000:82:00.0: irq 114 for MSI/MSI-X [ 13.571364] mlx4_core 0000:82:00.0: irq 115 for MSI/MSI-X [ 13.571383] mlx4_core 0000:82:00.0: irq 116 for MSI/MSI-X [ 13.571403] mlx4_core 0000:82:00.0: irq 117 for MSI/MSI-X [ 13.571422] mlx4_core 0000:82:00.0: irq 118 for MSI/MSI-X [ 13.665357] scsi 1:0:0:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 13.674462] scsi 1:0:0:2: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) [ 13.686205] scsi 1:0:0:2: enclosure logical id(0x500605b00aafc090), slot(3) [ 13.694069] scsi 1:0:0:2: enclosure level(0x0000), connector name( ) [ 13.701549] scsi 1:0:0:2: serial_number(021546008063 ) [ 13.707572] scsi 1:0:0:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 13.735206] scsi 1:0:0:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 13.744422] scsi 1:0:0:31: SSP: handle(0x0009), sas_addr(0x500a09849de54f10), phy(0), device_name(0x500a09849de54f10) [ 13.756261] scsi 1:0:0:31: enclosure logical id(0x500605b00aafc090), slot(3) [ 13.764221] scsi 1:0:0:31: enclosure level(0x0000), connector name( ) [ 13.771799] scsi 1:0:0:31: serial_number(021546008063 ) [ 13.777923] scsi 1:0:0:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 13.802546] mpt3sas_cm0: detecting: handle(0x000a), sas_address(0x500a09849dcf7b10), phy(4) [ 13.811873] mpt3sas_cm0: REPORT_LUNS: handle(0x000a), retries(0) [ 13.818725] mpt3sas_cm0: REPORT_LUNS: handle(0x000a), retries(1) [ 13.826447] mpt3sas_cm0: TEST_UNIT_READY: handle(0x000a), lun(0) [ 13.833469] mpt3sas_cm0: detecting: handle(0x000a), sas_address(0x500a09849dcf7b10), phy(4) [ 13.842795] mpt3sas_cm0: REPORT_LUNS: handle(0x000a), retries(0) [ 13.850117] mpt3sas_cm0: TEST_UNIT_READY: handle(0x000a), lun(0) [ 13.857720] scsi 1:0:1:0: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 13.866827] scsi 1:0:1:0: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) [ 13.878571] scsi 1:0:1:0: enclosure logical id(0x500605b00aafc090), slot(7) [ 13.886436] scsi 1:0:1:0: enclosure level(0x0000), connector name( ) [ 13.893921] scsi 1:0:1:0: serial_number(021546007824 ) [ 13.899949] scsi 1:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 13.929005] scsi 1:0:1:1: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 13.938122] scsi 1:0:1:1: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) [ 13.949869] scsi 1:0:1:1: enclosure logical id(0x500605b00aafc090), slot(7) [ 13.957735] scsi 1:0:1:1: enclosure level(0x0000), connector name( ) [ 13.965223] scsi 1:0:1:1: serial_number(021546007824 ) [ 13.971252] scsi 1:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 13.995325] scsi 1:0:1:2: Direct-Access DELL MD34xx 0825 PQ: 0 ANSI: 5 [ 14.004732] scsi 1:0:1:2: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) [ 14.016480] scsi 1:0:1:2: enclosure logical id(0x500605b00aafc090), slot(7) [ 14.024346] scsi 1:0:1:2: enclosure level(0x0000), connector name( ) [ 14.031835] scsi 1:0:1:2: serial_number(021546007824 ) [ 14.037864] scsi 1:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 14.065341] scsi 1:0:1:31: Direct-Access DELL Universal Xport 0825 PQ: 0 ANSI: 5 [ 14.074559] scsi 1:0:1:31: SSP: handle(0x000a), sas_addr(0x500a09849dcf7b10), phy(4), device_name(0x500a09849dcf7b10) [ 14.086402] scsi 1:0:1:31: enclosure logical id(0x500605b00aafc090), slot(7) [ 14.094366] scsi 1:0:1:31: enclosure level(0x0000), connector name( ) [ 14.101950] scsi 1:0:1:31: serial_number(021546007824 ) [ 14.108076] scsi 1:0:1:31: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 14.134960] mpt3sas_cm0: port enable: SUCCESS [ 14.140512] scsi 1:0:0:0: rdac: LUN 0 (IOSHIP) (owned) [ 14.146752] scsi 1:0:0:1: rdac: LUN 1 (IOSHIP) (unowned) [ 14.153220] scsi 1:0:0:2: rdac: LUN 2 (IOSHIP) (owned) [ 14.159735] scsi 1:0:1:0: rdac: LUN 0 (IOSHIP) (unowned) [ 14.166126] scsi 1:0:1:1: rdac: LUN 1 (IOSHIP) (owned) [ 14.172343] sd 0:2:0:0: [sda] 487325696 512-byte logical blocks: (249 GB/232 GiB) [ 14.172452] scsi 1:0:1:2: rdac: LUN 2 (IOSHIP) (unowned) [ 14.172498] sd 1:0:0:0: [sdb] 574402560 512-byte logical blocks: (294 GB/273 GiB) [ 14.172723] sd 1:0:0:1: [sdc] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) [ 14.172819] sd 1:0:1:0: [sdf] 574402560 512-byte logical blocks: (294 GB/273 GiB) [ 14.172824] sd 1:0:0:2: [sdd] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) [ 14.172860] sd 1:0:1:2: [sde] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) [ 14.173060] sd 1:0:1:1: [sdg] 3493273600 512-byte logical blocks: (1.78 TB/1.62 TiB) [ 14.173523] sd 1:0:1:0: [sdf] Write Protect is off [ 14.173526] sd 1:0:1:0: [sdf] Mode Sense: 83 00 10 08 [ 14.173618] sd 1:0:1:2: [sde] Write Protect is off [ 14.173620] sd 1:0:1:2: [sde] Mode Sense: 83 00 10 08 [ 14.173827] sd 1:0:1:1: [sdg] Write Protect is off [ 14.173830] sd 1:0:1:1: [sdg] Mode Sense: 83 00 10 08 [ 14.173851] sd 1:0:1:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 14.173867] sd 1:0:1:2: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 14.174076] sd 1:0:0:0: [sdb] Write Protect is off [ 14.174079] sd 1:0:0:0: [sdb] Mode Sense: 83 00 10 08 [ 14.174159] sd 1:0:1:1: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 14.174336] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 14.174442] sd 1:0:0:1: [sdc] Write Protect is off [ 14.174445] sd 1:0:0:1: [sdc] Mode Sense: 83 00 10 08 [ 14.174468] sd 1:0:0:2: [sdd] Write Protect is off [ 14.174471] sd 1:0:0:2: [sdd] Mode Sense: 83 00 10 08 [ 14.174825] sd 1:0:0:2: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 14.174847] sd 1:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 14.177893] sd 1:0:1:1: [sdg] Attached SCSI disk [ 14.178215] sd 1:0:0:0: [sdb] Attached SCSI disk [ 14.178538] sd 1:0:1:2: [sde] Attached SCSI disk [ 14.178812] sd 1:0:1:0: [sdf] Attached SCSI disk [ 14.179772] sd 1:0:0:2: [sdd] Attached SCSI disk [ 14.180537] sd 1:0:0:1: [sdc] Attached SCSI disk [ 14.358201] sd 0:2:0:0: [sda] Write Protect is off [ 14.363552] sd 0:2:0:0: [sda] Mode Sense: 1f 00 10 08 [ 14.363605] sd 0:2:0:0: [sda] Write cache: disabled, read cache: disabled, supports DPO and FUA [ 14.391553] sda: sda1 sda2 [ 14.395135] sd 0:2:0:0: [sda] Attached SCSI disk [ 14.716585] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) [ 14.758238] random: crng init done [ 15.134155] systemd-journald[217]: Received SIGTERM from PID 1 (systemd). [ 15.611215] SELinux: Disabled at runtime. [ 15.615821] SELinux: Unregistering netfilter hooks [ 15.669943] type=1404 audit(1567172267.440:2): selinux=0 auid=4294967295 ses=4294967295 [ 15.895530] ip_tables: (C) 2000-2006 Netfilter Core Team [ 15.923424] systemd[1]: Inserted module 'ip_tables' [ 16.990349] EXT4-fs (sda1): re-mounted. Opts: (null) [ 17.190275] systemd-journald[2637]: Received request to flush runtime journal from PID 1 [ 17.297107] device-mapper: uevent: version 1.0.3 [ 17.302392] device-mapper: ioctl: 4.37.1-ioctl (2018-04-03) initialised: dm-devel@redhat.com [ 17.755657] ACPI Error: No handler for Region [SYSI] (ffff8e5aa9ea6870) [IPMI] (20130517/evregion-162) [ 17.766095] ACPI Error: Region IPMI (ID=7) has no handler (20130517/exfldio-305) [ 17.774404] ACPI Error: Method parse/execution failed [\_SB_.PMI0._GHL] (Node ffff8e4b29444550), AE_NOT_EXIST (20130517/psparse-536) [ 17.787796] ACPI Error: Method parse/execution failed [\_SB_.PMI0._PMC] (Node ffff8e4b294444b0), AE_NOT_EXIST (20130517/psparse-536) [ 17.801170] ACPI Exception: AE_NOT_EXIST, Evaluating _PMC (20130517/power_meter-753) [ 17.844207] ipmi message handler version 39.2 [ 17.850966] ipmi device interface [ 17.863060] IPMI System Interface driver [ 17.867455] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS [ 17.874556] ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 [ 17.881261] ipmi_si: Adding SMBIOS-specified kcs state machine [ 17.887851] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 [ 17.902727] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ 17.954987] ipmi_si dmi-ipmi-si.0: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed. [ 17.976052] ipmi_si dmi-ipmi-si.0: Using irq 10 [ 17.982298] RPC: Registered named UNIX socket transport module. [ 17.984574] ipmi_si dmi-ipmi-si.0: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) [ 17.992688] type=1305 audit(1567172269.762:3): audit_pid=2817 old=0 auid=4294967295 ses=4294967295 res=1 [ 18.009514] RPC: Registered udp transport module. [ 18.014774] RPC: Registered tcp transport module. [ 18.020047] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 18.042124] ipmi_si dmi-ipmi-si.0: IPMI kcs interface initialized [ 18.198882] sd 0:2:0:0: Attached scsi generic sg0 type 0 [ 18.205002] sd 1:0:0:0: Attached scsi generic sg1 type 0 [ 18.211165] sd 1:0:0:1: Attached scsi generic sg2 type 0 [ 18.217998] sd 1:0:0:2: Attached scsi generic sg3 type 0 [ 18.225918] scsi 1:0:0:31: Attached scsi generic sg4 type 0 [ 18.233225] sd 1:0:1:0: Attached scsi generic sg5 type 0 [ 18.239937] input: PC Speaker as /devices/platform/pcspkr/input/input1 [ 18.241458] sd 1:0:1:1: Attached scsi generic sg6 type 0 [ 18.241535] sd 1:0:1:2: Attached scsi generic sg7 type 0 [ 18.241613] scsi 1:0:1:31: Attached scsi generic sg8 type 0 [ 18.273669] cryptd: max_cpu_qlen set to 1000 [ 18.290366] AVX2 version of gcm_enc/dec engaged. [ 18.295560] AES CTR mode by8 optimization enabled [ 18.304091] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 18.311379] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni) [ 18.398733] intel_rapl: Found RAPL domain package [ 18.404004] intel_rapl: Found RAPL domain dram [ 18.409094] intel_rapl: DRAM domain energy unit 15300pj [ 18.414969] intel_rapl: RAPL package 0 domain package locked by BIOS [ 18.422204] intel_rapl: Found RAPL domain package [ 18.427513] intel_rapl: Found RAPL domain dram [ 18.432603] intel_rapl: DRAM domain energy unit 15300pj [ 18.438477] intel_rapl: RAPL package 1 domain package locked by BIOS [ 18.454884] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 [ 18.455306] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 [ 18.455319] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 [ 18.455325] EDAC sbridge: Seeking for: PCI ID 8086:2f60 [ 18.455333] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 [ 18.455338] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 [ 18.455343] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 [ 18.455345] EDAC sbridge: Seeking for: PCI ID 8086:2f71 [ 18.455350] EDAC sbridge: Seeking for: PCI ID 8086:2f71 [ 18.455355] EDAC sbridge: Seeking for: PCI ID 8086:2f71 [ 18.455357] EDAC sbridge: Seeking for: PCI ID 8086:2faa [ 18.455361] EDAC sbridge: Seeking for: PCI ID 8086:2faa [ 18.455367] EDAC sbridge: Seeking for: PCI ID 8086:2faa [ 18.455369] EDAC sbridge: Seeking for: PCI ID 8086:2fab [ 18.455376] EDAC sbridge: Seeking for: PCI ID 8086:2fab [ 18.455381] EDAC sbridge: Seeking for: PCI ID 8086:2fab [ 18.455383] EDAC sbridge: Seeking for: PCI ID 8086:2fac [ 18.455405] EDAC sbridge: Seeking for: PCI ID 8086:2fac [ 18.455412] EDAC sbridge: Seeking for: PCI ID 8086:2fac [ 18.455414] EDAC sbridge: Seeking for: PCI ID 8086:2fad [ 18.455419] EDAC sbridge: Seeking for: PCI ID 8086:2fad [ 18.455424] EDAC sbridge: Seeking for: PCI ID 8086:2fad [ 18.455426] EDAC sbridge: Seeking for: PCI ID 8086:2f68 [ 18.455431] EDAC sbridge: Seeking for: PCI ID 8086:2f79 [ 18.455439] EDAC sbridge: Seeking for: PCI ID 8086:2f6a [ 18.455445] EDAC sbridge: Seeking for: PCI ID 8086:2f6b [ 18.455453] EDAC sbridge: Seeking for: PCI ID 8086:2f6c [ 18.455460] EDAC sbridge: Seeking for: PCI ID 8086:2f6d [ 18.455467] EDAC sbridge: Seeking for: PCI ID 8086:2ffc [ 18.455476] EDAC sbridge: Seeking for: PCI ID 8086:2ffc [ 18.455481] EDAC sbridge: Seeking for: PCI ID 8086:2ffc [ 18.455485] EDAC sbridge: Seeking for: PCI ID 8086:2ffd [ 18.455490] EDAC sbridge: Seeking for: PCI ID 8086:2ffd [ 18.455496] EDAC sbridge: Seeking for: PCI ID 8086:2ffd [ 18.455499] EDAC sbridge: Seeking for: PCI ID 8086:2fbd [ 18.455504] EDAC sbridge: Seeking for: PCI ID 8086:2fbd [ 18.455510] EDAC sbridge: Seeking for: PCI ID 8086:2fbd [ 18.455512] EDAC sbridge: Seeking for: PCI ID 8086:2fbf [ 18.455519] EDAC sbridge: Seeking for: PCI ID 8086:2fbf [ 18.455525] EDAC sbridge: Seeking for: PCI ID 8086:2fbf [ 18.455527] EDAC sbridge: Seeking for: PCI ID 8086:2fb9 [ 18.455532] EDAC sbridge: Seeking for: PCI ID 8086:2fb9 [ 18.455537] EDAC sbridge: Seeking for: PCI ID 8086:2fb9 [ 18.455540] EDAC sbridge: Seeking for: PCI ID 8086:2fbb [ 18.455563] EDAC sbridge: Seeking for: PCI ID 8086:2fbb [ 18.455570] EDAC sbridge: Seeking for: PCI ID 8086:2fbb [ 18.455728] EDAC MC0: Giving out device to 'sb_edac.c' 'Haswell SrcID#0_Ha#0': DEV 0000:7f:12.0 [ 18.466542] EDAC MC1: Giving out device to 'sb_edac.c' 'Haswell SrcID#1_Ha#0': DEV 0000:ff:12.0 [ 18.476275] EDAC sbridge: Ver: 1.1.2 [ 18.516146] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.3) [ 18.525572] iTCO_vendor_support: vendor-support=0 [ 18.532099] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 18.538351] iTCO_wdt: Found a Wellsburg TCO device (Version=2, TCOBASE=0x0460) [ 18.546489] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) [ 18.700183] device-mapper: multipath round-robin: version 1.2.0 loaded [ 19.954179] device-mapper: multipath service-time: version 0.3.0 loaded [ 19.961992] device-mapper: table: 253:3: multipath: error getting device [ 19.969480] device-mapper: ioctl: error adding target to table [ 20.283601] Adding 4194300k swap on /dev/sda2. Priority:-2 extents:1 across:4194300k FS [ 22.657066] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.5-1.0.1 [ 22.666928] <mlx4_ib> mlx4_ib_add: counter index 0 for port 1 allocated 0 [ 23.015197] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.5-1.0.1 [ 23.834328] card: mlx4_0, QP: 0x220, inline size: 120 [ 24.596883] tg3 0000:01:00.0: irq 119 for MSI/MSI-X [ 24.596950] tg3 0000:01:00.0: irq 120 for MSI/MSI-X [ 24.596991] tg3 0000:01:00.0: irq 121 for MSI/MSI-X [ 24.597029] tg3 0000:01:00.0: irq 122 for MSI/MSI-X [ 24.597094] tg3 0000:01:00.0: irq 123 for MSI/MSI-X [ 24.721544] IPv6: ADDRCONF(NETDEV_UP): em1: link is not ready [ 28.237568] tg3 0000:01:00.0 em1: Link is up at 1000 Mbps, full duplex [ 28.244863] tg3 0000:01:00.0 em1: Flow control is off for TX and off for RX [ 28.252636] tg3 0000:01:00.0 em1: EEE is enabled [ 28.257824] IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready [ 29.078393] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 29.086861] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 33.357318] FS-Cache: Loaded [ 33.406821] FS-Cache: Netfs 'nfs' registered for caching [ 33.417300] Key type dns_resolver registered [ 33.454336] NFS: Registering the id_resolver key type [ 33.459978] Key type id_resolver registered [ 33.464647] Key type id_legacy registered [ 52.193854] LNet: HW NUMA nodes: 2, HW CPU cores: 24, npartitions: 2 [ 52.201982] alg: No test for adler32 (adler32-zlib) [ 52.977457] Lustre: Lustre: Build Version: 2.10.8_3_g73a88a8 [ 53.117716] LNet: Added LNI 10.0.2.52@o2ib5 [8/256/0/180] [ 60.891958] LNetError: 2515:0:(o2iblnd_cb.c:2309:kiblnd_passive_connect()) Can't accept conn from 10.0.2.202@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 [ 61.516254] LNetError: 2515:0:(o2iblnd_cb.c:2309:kiblnd_passive_connect()) Can't accept conn from 10.0.2.249@o2ib5 on NA (ib0:1:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 [ 61.532951] LNetError: 2515:0:(o2iblnd_cb.c:2309:kiblnd_passive_connect()) Skipped 5 previous similar messages [ 62.881175] LNet: Removed LNI 10.0.2.52@o2ib5 [ 3141.643897] LNet: HW NUMA nodes: 2, HW CPU cores: 24, npartitions: 2 [ 3141.651976] alg: No test for adler32 (adler32-zlib) [ 3142.432410] Lustre: Lustre: Build Version: 2.10.8_3_g73a88a8 [ 3142.522981] LNetError: 2515:0:(o2iblnd_cb.c:2309:kiblnd_passive_connect()) Can't accept conn from 10.0.2.252@o2ib5 on NA (ib0:0:10.0.2.52): bad dst nid 10.0.2.52@o2ib5 [ 3142.527323] LNet: Added LNI 10.0.2.52@o2ib5 [8/256/0/180] [ 3245.442471] LDISKFS-fs warning (device dm-1): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. [ 3927.481032] LDISKFS-fs (dm-1): recovery complete [ 3927.486597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,acl,no_mbcache,nodelalloc [ 3931.713841] LustreError: 137-5: oak-MDT0000_UUID: not available for connect from 10.8.22.35@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.733155] LustreError: Skipped 1 previous similar message [ 3932.258284] Lustre: oak-MDT0000: Not available for connect from 10.8.19.2@o2ib6 (not set up) [ 3939.496578] LustreError: 11-0: oak-MDT0001-osp-MDT0000: operation mds_connect to node 10.0.2.51@o2ib5 failed: rc = -114 [ 3940.761058] Lustre: oak-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 3941.035645] Lustre: oak-MDD0000: changelog on [ 3941.179130] Lustre: oak-MDT0000: in recovery but waiting for the first client to connect [ 3941.322810] Lustre: oak-MDT0000: Will be in recovery for at least 2:30, or until 1479 clients reconnect [ 3941.333324] Lustre: oak-MDT0000: Connection restored to 7be46174-537c-ddf8-d8a4-d36639de207d (at 10.8.12.17@o2ib6) [ 3941.854031] Lustre: oak-MDT0000: Connection restored to 9cefe124-9e8b-981c-aa16-921fdf8695d5 (at 10.9.113.2@o2ib4) [ 3941.865589] Lustre: Skipped 12 previous similar messages [ 3943.027083] Lustre: oak-MDT0000: Connection restored to 10.0.2.113@o2ib5 (at 10.0.2.113@o2ib5) [ 3943.036701] Lustre: Skipped 10 previous similar messages [ 3943.611328] Lustre: oak-MDT0000: nosquash_nids set to 10.0.2.[1-3]@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[66-76]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.210.47.253@o2ib3 10.9.0.[1-2]@o2ib4 10.9.101.[59-60]@o2ib4 10.8.2.[1-34]@o2ib6 10.8.03.[1-36]@o2ib6 10.8.4.[01-36]@o2ib6 10.8.6.[01-36]@o2ib6 10.8.9.[1-2,6]@o2ib6 10.8.10.[1-36]@o2ib6 10.8.11.[1-36]@o2ib6 10.8.12.[1-36]@o2ib6 10.8.13.[1-22]@o2ib6 10.0.2.113@o2ib5 171.67.92.[38,53]@tcp1 [ 3943.726255] Lustre: oak-MDT0000: root_squash is set to 99:99 [ 3945.110231] Lustre: oak-MDT0000: Connection restored to e9accd7d-bf5c-c4c2-b394-1836e5f3fe59 (at 10.8.3.18@o2ib6) [ 3945.121693] Lustre: Skipped 65 previous similar messages [ 3949.111785] Lustre: oak-MDT0000: Connection restored to 099d79dd-72e9-80df-2aff-bd342fe2c255 (at 10.8.22.31@o2ib6) [ 3949.123347] Lustre: Skipped 371 previous similar messages [ 3957.132796] Lustre: oak-MDT0000: Connection restored to 9312c385-cb4b-3de0-a108-139cb71e5574 (at 10.8.7.18@o2ib6) [ 3957.144259] Lustre: Skipped 353 previous similar messages [ 3965.759268] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3965.777420] LustreError: Skipped 15 previous similar messages [ 3973.255045] Lustre: oak-MDT0000: Connection restored to 2f60bd96-9546-4be2-05c1-b7a5d934b1e1 (at 10.8.28.9@o2ib6) [ 3973.266505] Lustre: Skipped 501 previous similar messages [ 3990.491366] LustreError: 16864:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8e696260e000 x1643302473498592/t0(0) o601->oak-MDT0000-lwp-OST0028_UUID@10.0.2.101@o2ib5:118/0 lens 336/0 e 0 to 0 dl 1567176268 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 3990.520282] LustreError: 16864:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 1 previous similar message [ 4016.305662] Lustre: oak-MDT0000: Connection restored to 10.0.2.110@o2ib5 (at 10.0.2.110@o2ib5) [ 4016.305663] Lustre: oak-MDT0000: Connection restored to 10.0.2.110@o2ib5 (at 10.0.2.110@o2ib5) [ 4016.305665] Lustre: Skipped 259 previous similar messages [ 4016.330916] Lustre: Skipped 19 previous similar messages [ 4016.502632] LustreError: 16929:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8e58c7d24b00 x1643302473484688/t0(0) o601->oak-MDT0000-lwp-OST007d_UUID@10.0.2.110@o2ib5:144/0 lens 336/0 e 0 to 0 dl 1567176294 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 4050.491456] LustreError: 16857:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8e6958327500 x1643302473513376/t0(0) o601->oak-MDT0000-lwp-OST0028_UUID@10.0.2.101@o2ib5:178/0 lens 336/0 e 0 to 0 dl 1567176328 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 4050.520370] LustreError: 16857:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 1 previous similar message [ 4076.503503] LustreError: 16927:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8e58d35a6450 x1643302473506368/t0(0) o601->oak-MDT0000-lwp-OST007d_UUID@10.0.2.110@o2ib5:204/0 lens 336/0 e 0 to 0 dl 1567176354 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 4091.593548] Lustre: oak-MDT0000: recovery is timed out, evict stale exports [ 4091.601458] Lustre: oak-MDT0000: disconnecting 2 stale clients [ 4095.086756] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4096.139043] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4096.149046] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3 previous similar messages [ 4097.140169] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4097.150175] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 18 previous similar messages [ 4099.403292] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4099.413298] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 9 previous similar messages [ 4103.583554] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4103.593557] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 19 previous similar messages [ 4110.492299] LustreError: 16857:0:(tgt_handler.c:509:tgt_filter_recovery_request()) @@@ not permitted during recovery req@ffff8e6955a79200 x1643302473514944/t0(0) o601->oak-MDT0000-lwp-OST0028_UUID@10.0.2.101@o2ib5:238/0 lens 336/0 e 0 to 0 dl 1567176388 ref 1 fl Interpret:/0/ffffffff rc 0/-1 [ 4110.521211] LustreError: 16857:0:(tgt_handler.c:509:tgt_filter_recovery_request()) Skipped 1 previous similar message [ 4111.637528] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4111.647538] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 69 previous similar messages [ 4127.644288] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4127.654292] LustreError: 16578:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 413 previous similar messages [ 4136.528540] Lustre: oak-MDT0000: Recovery over after 3:15, of 1479 clients 1477 recovered and 2 were evicted. [ 4159.658264] LustreError: 16926:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4159.668268] LustreError: 16926:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 4383 previous similar messages [ 4224.610181] LustreError: 16926:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4224.620188] LustreError: 16926:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 890 previous similar messages [ 4313.935482] Lustre: oak-MDT0000: Connection restored to 2b1b65e7-505a-46e9-55bc-2bab38bdb11c (at 10.9.105.5@o2ib4) [ 4313.947041] Lustre: Skipped 48 previous similar messages [ 4353.172971] LustreError: 16873:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4353.182971] LustreError: 16873:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1409 previous similar messages [ 4363.009772] Lustre: oak-MDT0000: haven't heard from client 2b1b65e7-505a-46e9-55bc-2bab38bdb11c (at 10.9.105.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e696680a000, cur 1567176615 expire 1567176465 last 1567176388 [ 4609.230158] LustreError: 16853:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 4609.240166] LustreError: 16853:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 13848 previous similar messages [ 5121.272180] LustreError: 33402:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 5121.282187] LustreError: 33402:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 48928 previous similar messages [ 5336.142718] Lustre: oak-MDT0000: Connection restored to cc3eaec9-3f0c-a83b-fd34-c74d87bb3b3d (at 10.0.2.3@o2ib5) [ 5721.283342] LustreError: 33402:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 5721.293346] LustreError: 33402:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 89070 previous similar messages [ 6321.313636] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 6321.323641] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 245491 previous similar messages [ 6686.001801] Lustre: oak-MDT0000: haven't heard from client 27afcda6-6908-af01-93a4-42c930808ce9 (at 10.9.0.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966f97000, cur 1567178938 expire 1567178788 last 1567178711 [ 6686.025871] Lustre: Skipped 1 previous similar message [ 6921.644654] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 6921.654664] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 214882 previous similar messages [ 7247.766635] Lustre: 33152:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [ 7316.881904] Lustre: oak-MDT0000: Connection restored to (at 10.9.0.1@o2ib4) [ 7521.803309] LustreError: 16602:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 7521.813339] LustreError: 16602:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 73264 previous similar messages [ 7552.500458] Lustre: 33081:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [ 7552.513469] Lustre: 33081:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [ 7584.639733] Lustre: 33117:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [ 7584.652760] Lustre: 33117:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages [ 8121.854365] LustreError: 16862:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 8121.864386] LustreError: 16862:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 55303 previous similar messages [ 8236.447884] Lustre: oak-MDT0000: Client 44acd0ef-cd10-88b5-d309-b93efc9df9b8 (at 10.0.2.228@o2ib5) reconnecting [ 8236.459168] Lustre: oak-MDT0000: Connection restored to 44acd0ef-cd10-88b5-d309-b93efc9df9b8 (at 10.0.2.228@o2ib5) [ 8335.596806] LNet: 33527:0:(o2iblnd_cb.c:2415:kiblnd_passive_connect()) Conn stale 10.0.2.250@o2ib5 version 12/12 incarnation 1551985169829937/1567180586181763 [ 8335.622860] Lustre: oak-MDT0000: Connection restored to b456c0f5-dcda-cc59-3c00-8ebc01f45327 (at 10.0.2.250@o2ib5) [ 8461.038934] Lustre: oak-MDT0000: haven't heard from client b456c0f5-dcda-cc59-3c00-8ebc01f45327 (at 10.0.2.250@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966e0e400, cur 1567180713 expire 1567180563 last 1567180486 [ 8537.033428] Lustre: oak-MDT0000: haven't heard from client 1d747c02-9882-7c86-0e7c-d60d12760dee (at 10.0.2.250@o2ib5) in 202 seconds. I think it's dead, and I am evicting it. exp ffff8e692c4f4400, cur 1567180789 expire 1567180639 last 1567180587 [ 8537.392398] Lustre: oak-MDT0000: Client 44acd0ef-cd10-88b5-d309-b93efc9df9b8 (at 10.0.2.228@o2ib5) reconnecting [ 8537.403674] Lustre: oak-MDT0000: Connection restored to 44acd0ef-cd10-88b5-d309-b93efc9df9b8 (at 10.0.2.228@o2ib5) [ 8590.135563] Lustre: oak-MDT0000: Connection restored to b456c0f5-dcda-cc59-3c00-8ebc01f45327 (at 10.0.2.250@o2ib5) [ 8721.955081] LustreError: 16927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 8721.965087] LustreError: 16927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 47168 previous similar messages [ 8998.918103] Lustre: oak-MDT0000: Client 44acd0ef-cd10-88b5-d309-b93efc9df9b8 (at 10.0.2.228@o2ib5) reconnecting [ 8998.929409] Lustre: oak-MDT0000: Connection restored to 44acd0ef-cd10-88b5-d309-b93efc9df9b8 (at 10.0.2.228@o2ib5) [ 9012.038184] Lustre: oak-MDT0000: haven't heard from client 8737b60c-3010-1ae1-5a8a-cb7ce80262f8 (at 10.12.4.59@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e696696c800, cur 1567181264 expire 1567181114 last 1567181037 [ 9322.017432] LustreError: 16883:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 9322.027439] LustreError: 16883:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 34092 previous similar messages [ 9728.049690] Lustre: oak-MDT0000: haven't heard from client 56fec540-e9f5-e031-f272-03d5e4588620 (at 10.8.18.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966969000, cur 1567181980 expire 1567181830 last 1567181753 [ 9781.410460] LustreError: 11-0: oak-OST0055-osc-MDT0000: operation ost_statfs to node 10.0.2.106@o2ib5 failed: rc = -107 [ 9781.422507] Lustre: oak-OST0055-osc-MDT0000: Connection to oak-OST0055 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [ 9782.258437] LustreError: 11-0: oak-OST005d-osc-MDT0000: operation ost_statfs to node 10.0.2.106@o2ib5 failed: rc = -107 [ 9782.270482] LustreError: Skipped 1 previous similar message [ 9782.276704] Lustre: oak-OST005d-osc-MDT0000: Connection to oak-OST005d (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [ 9782.294569] Lustre: Skipped 1 previous similar message [ 9783.266344] LustreError: 11-0: oak-OST0033-osc-MDT0000: operation ost_statfs to node 10.0.2.106@o2ib5 failed: rc = -107 [ 9783.278388] LustreError: Skipped 8 previous similar messages [ 9783.284705] Lustre: oak-OST0033-osc-MDT0000: Connection to oak-OST0033 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [ 9783.302563] Lustre: Skipped 10 previous similar messages [ 9791.722372] Lustre: 15787:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567182036/real 1567182036] req@ffff8e5600d1b600 x1643302542518400/t0(0) o13->oak-OST0049-osc-MDT0000@10.0.2.106@o2ib5:7/4 lens 224/368 e 0 to 1 dl 1567182043 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [ 9791.753710] Lustre: oak-OST0049-osc-MDT0000: Connection to oak-OST0049 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [ 9791.771563] Lustre: Skipped 9 previous similar messages [ 9791.779153] LNetError: 33527:0:(o2iblnd_cb.c:2769:kiblnd_rejected()) 10.0.2.106@o2ib5 rejected: o2iblnd fatal error [ 9828.173147] LustreError: 11-0: oak-OST005a-osc-MDT0000: operation ost_destroy to node 10.0.2.105@o2ib5 failed: rc = -19 [ 9828.185190] LustreError: Skipped 11 previous similar messages [ 9828.191618] Lustre: oak-OST005a-osc-MDT0000: Connection to oak-OST005a (at 10.0.2.105@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [ 9832.659113] LustreError: 11-0: oak-OST0030-osc-MDT0000: operation ost_statfs to node 10.0.2.105@o2ib5 failed: rc = -107 [ 9832.671151] LustreError: Skipped 17 previous similar messages [ 9852.778299] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567182093/real 0] req@ffff8e690f69e000 x1643302542549264/t0(0) o8->oak-OST005f-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567182104 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9852.808859] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9858.172442] Lustre: oak-MDT0000: Client a8523b5a-875e-d012-8fb3-3ddb42266176 (at 10.9.104.28@o2ib4) reconnecting [ 9858.183814] Lustre: oak-MDT0000: Connection restored to a8523b5a-875e-d012-8fb3-3ddb42266176 (at 10.9.104.28@o2ib4) [ 9863.254959] Lustre: oak-MDT0000: Client 42dc9860-d970-704b-ba7a-1bdee4709df7 (at 10.8.26.8@o2ib6) reconnecting [ 9863.266138] Lustre: oak-MDT0000: Connection restored to 42dc9860-d970-704b-ba7a-1bdee4709df7 (at 10.8.26.8@o2ib6) [ 9864.389223] Lustre: oak-MDT0000: Client 531a3829-cf82-d8fb-60d8-447a2f9ef900 (at 10.9.104.22@o2ib4) reconnecting [ 9866.691814] Lustre: oak-MDT0000: Client 8a2001d1-6381-f16e-8959-be8befc8ab3e (at 10.8.20.29@o2ib6) reconnecting [ 9866.703084] Lustre: oak-MDT0000: Connection restored to 8a2001d1-6381-f16e-8959-be8befc8ab3e (at 10.8.20.29@o2ib6) [ 9866.714636] Lustre: Skipped 1 previous similar message [ 9872.778648] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567182118/real 0] req@ffff8e690d67e900 x1643302542559968/t0(0) o8->oak-OST005e-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567182124 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9872.809211] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 9874.770048] Lustre: oak-MDT0000: Client 73850156-6406-43df-c6b4-806babd3151f (at 10.9.103.44@o2ib4) reconnecting [ 9874.781494] Lustre: Skipped 7 previous similar messages [ 9874.787358] Lustre: oak-MDT0000: Connection restored to 73850156-6406-43df-c6b4-806babd3151f (at 10.9.103.44@o2ib4) [ 9874.799034] Lustre: Skipped 7 previous similar messages [ 9887.829717] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.8.25.14@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9888.469888] Lustre: oak-MDT0000: Client 527efb28-29f9-1922-90b0-d1bdc5f3b7d6 (at 10.9.110.23@o2ib4) reconnecting [ 9888.481249] Lustre: Skipped 6 previous similar messages [ 9888.487099] Lustre: oak-MDT0000: Connection restored to 527efb28-29f9-1922-90b0-d1bdc5f3b7d6 (at 10.9.110.23@o2ib4) [ 9888.498749] Lustre: Skipped 6 previous similar messages [ 9888.983523] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.8.22.2@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.002752] LustreError: Skipped 1 previous similar message [ 9890.765050] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.8.17.25@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.007428] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.8.17.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.026741] LustreError: Skipped 1 previous similar message [ 9904.200067] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.8.0.67@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9904.219286] LustreError: Skipped 1 previous similar message [ 9907.779199] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567182143/real 1567182153] req@ffff8e690d67aa00 x1643302542570288/t0(0) o8->oak-OST003f-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567182159 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9907.810634] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [ 9915.803282] Lustre: oak-MDT0000: Client af9f23e0-23a0-d49b-c5b3-56588e0b33a9 (at 10.8.18.31@o2ib6) reconnecting [ 9915.814544] Lustre: Skipped 1 previous similar message [ 9915.820287] Lustre: oak-MDT0000: Connection restored to (at 10.8.18.31@o2ib6) [ 9915.828353] Lustre: Skipped 1 previous similar message [ 9916.671347] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 171.67.92.36@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9922.566097] LustreError: 16618:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [ 9922.576105] LustreError: 16618:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 13779 previous similar messages [ 9927.779509] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567182168/real 0] req@ffff8e690b621b00 x1643302542581280/t0(0) o8->oak-OST005f-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567182179 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9927.810072] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [ 9936.930388] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 171.65.177.203@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9948.754244] Lustre: oak-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [ 9948.763862] Lustre: Skipped 16 previous similar messages [ 9957.166048] Lustre: oak-MDT0000: Client dc235a4a-b343-fd2d-31fd-1f7fd06f7a95 (at 10.9.102.8@o2ib4) reconnecting [ 9957.177317] Lustre: Skipped 8 previous similar messages [ 9991.781160] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1567182243/real 1567182243] req@ffff8e692cc81b00 x1643302542614496/t0(0) o8->oak-OST0030-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567182259 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [ 9991.812980] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 47 previous similar messages [10013.078220] Lustre: oak-MDT0000: haven't heard from client 1ffa416a-b4e5-abfa-df15-020cc1c31fd1 (at 10.9.101.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e69674e7000, cur 1567182265 expire 1567182115 last 1567182038 [10039.264488] Lustre: oak-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [10039.274105] Lustre: Skipped 17 previous similar messages [10041.781989] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1567182293/real 1567182293] req@ffff8e692cc84200 x1643302542636688/t0(0) o8->oak-OST003e-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567182314 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [10041.813814] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [10167.859101] Lustre: oak-OST0045-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [10167.869889] Lustre: Skipped 26 previous similar messages [10250.385447] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [10250.400696] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567182202, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e692e2c8200/0x8223d61206a201be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d56c176d48 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [10250.401056] LustreError: 34240:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692e670f00) refcount nonzero (2) after lock cleanup; forcing cleanup. [10250.401073] LustreError: 34240:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692e670f00) refcount = 3 [10250.401076] LustreError: 34240:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [10250.401080] LustreError: 34240:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e692e2c8200/0x8223d61206a201be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d56c176d48 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [10324.501564] Lustre: 33222:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10336.653687] Lustre: 33141:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10336.666693] Lustre: 33141:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [10349.030861] Lustre: 33081:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10349.043871] Lustre: 33081:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages [10412.510612] Lustre: 16600:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10412.523634] Lustre: 16600:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages [10514.287449] Lustre: 33117:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10522.734651] LustreError: 16930:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [10522.744654] LustreError: 16930:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3808 previous similar messages [10537.088312] Lustre: 33194:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10537.101323] Lustre: 33194:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages [10554.117038] Lustre: 33079:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10554.130047] Lustre: 33079:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 24 previous similar messages [10559.506094] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [10559.521358] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567182511, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e54ee03ce00/0x8223d61206a8b44e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d56e107b46 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [10559.534307] LustreError: 34328:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692e7eb680) refcount nonzero (2) after lock cleanup; forcing cleanup. [10559.534314] LustreError: 34328:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692e7eb680) refcount = 3 [10559.534317] LustreError: 34328:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [10559.534329] LustreError: 34328:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e54ee03ce00/0x8223d61206a8b44e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d56e107b46 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [10559.534338] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [10559.534341] Lustre: Skipped 45 previous similar messages [10619.491016] Lustre: 33245:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10692.474191] Lustre: 33183:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [10692.512540] Lustre: 33183:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 36 previous similar messages [10775.649410] Lustre: 15799:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567182426/real 1567182426] req@ffff8e6958d8bc00 x1643302542693376/t0(0) o6->oak-OST005b-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 20 to 1 dl 1567183027 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [10775.742102] Lustre: 15799:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [10775.774444] Lustre: oak-OST005b-osc-MDT0000: Connection to oak-OST005b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [10775.827100] Lustre: Skipped 23 previous similar messages [10865.882811] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [10865.927741] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567182817, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5483884000/0x8223d6120794d561 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d57016c004 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [10865.927996] LustreError: 34373:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6932619c80) refcount nonzero (2) after lock cleanup; forcing cleanup. [10865.927999] LustreError: 34373:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6932619c80) refcount = 3 [10865.928001] LustreError: 34373:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [10865.928005] LustreError: 34373:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5483884000/0x8223d6120794d561 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d57016c004 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [11123.638909] LustreError: 16606:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [11123.746155] LustreError: 16606:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 4403 previous similar messages [11175.755639] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [11175.919051] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567183127, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e539cc84000/0x8223d61208b6117e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d572462508 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [11175.919289] LustreError: 34409:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692d6eec00) refcount nonzero (2) after lock cleanup; forcing cleanup. [11175.919292] LustreError: 34409:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692d6eec00) refcount = 3 [11175.919294] LustreError: 34409:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [11175.919299] LustreError: 34409:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e539cc84000/0x8223d61208b6117e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d572462508 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [11175.919304] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [11175.919305] Lustre: Skipped 2 previous similar messages [11224.994194] Lustre: 33184:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [11225.133676] Lustre: 33184:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 109 previous similar messages [11251.078631] Lustre: oak-MDT0000: haven't heard from client acf72813-8040-f3c0-c087-3f50a04c9df4 (at 10.9.109.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e69665b1800, cur 1567183503 expire 1567183353 last 1567183276 [11376.930719] Lustre: 15799:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567183027/real 1567183027] req@ffff8e6958d8bc00 x1643302542693376/t0(0) o6->oak-OST005b-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 20 to 1 dl 1567183628 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [11377.267807] Lustre: oak-OST005b-osc-MDT0000: Connection to oak-OST005b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [11485.001387] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [11485.164812] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567183436, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52c691cc00/0x8223d61208b9eb7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5745c9d5b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [11485.165053] LustreError: 34498:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692ea3a300) refcount nonzero (2) after lock cleanup; forcing cleanup. [11485.165055] LustreError: 34498:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692ea3a300) refcount = 3 [11485.165057] LustreError: 34498:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [11485.165061] LustreError: 34498:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52c691cc00/0x8223d61208b9eb7b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5745c9d5b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [11487.467485] Lustre: 33243:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [11487.606962] Lustre: 33243:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 21 previous similar messages [11724.128886] LustreError: 34447:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [11724.236139] LustreError: 34447:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 16708 previous similar messages [11792.726167] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [11792.889578] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567183744, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e692c167a00/0x8223d61209c09964 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d576d5104b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [11792.889713] LustreError: 34538:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692f62a9c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [11792.889716] LustreError: 34538:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692f62a9c0) refcount = 3 [11792.889717] LustreError: 34538:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [11792.889722] LustreError: 34538:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e692c167a00/0x8223d61209c09964 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d576d5104b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [11792.889726] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [11792.889727] Lustre: Skipped 2 previous similar messages [11978.220034] Lustre: 15799:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567183629/real 1567183629] req@ffff8e6958d8bc00 x1643302542693376/t0(0) o6->oak-OST005b-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 664/432 e 20 to 1 dl 1567184230 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [11978.557129] Lustre: oak-OST005b-osc-MDT0000: Connection to oak-OST005b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [12104.121979] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [12104.285397] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567184056, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e58b2cfbc00/0x8223d6120b04b100 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5798f6807 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [12104.285678] LustreError: 34586:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e69303198c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [12104.285685] LustreError: 34586:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e69303198c0) refcount = 3 [12104.285688] LustreError: 34586:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [12104.285701] LustreError: 34586:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e58b2cfbc00/0x8223d6120b04b100 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5798f6807 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [12324.794045] LustreError: 34451:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [12324.901288] LustreError: 34451:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 9796 previous similar messages [12333.310387] LustreError: 11-0: oak-OST003c-osc-MDT0000: operation ost_destroy to node 10.0.2.106@o2ib5 failed: rc = -19 [12333.317268] Lustre: oak-OST0038-osc-MDT0000: Connection to oak-OST0038 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [12333.630945] LustreError: Skipped 11 previous similar messages [12334.441736] LustreError: 11-0: oak-OST004c-osc-MDT0000: operation ost_statfs to node 10.0.2.106@o2ib5 failed: rc = -107 [12334.570829] LustreError: Skipped 2 previous similar messages [12336.985689] LustreError: 11-0: oak-OST0030-osc-MDT0000: operation ost_statfs to node 10.0.2.106@o2ib5 failed: rc = -107 [12337.114769] LustreError: Skipped 6 previous similar messages [12385.289363] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567184631/real 0] req@ffff8e692deaec00 x1643302562455952/t0(0) o8->oak-OST005e-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567184637 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12407.202728] Lustre: oak-MDT0000: haven't heard from client 44acd0ef-cd10-88b5-d309-b93efc9df9b8 (at 10.0.2.228@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966c3d800, cur 1567184659 expire 1567184509 last 1567184432 [12413.142779] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [12413.306193] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567184365, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51b2c9c600/0x8223d6120c4a96db lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d57c3b1e29 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [12413.306543] LustreError: 34650:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6922fc12c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [12413.306549] LustreError: 34650:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6922fc12c0) refcount = 3 [12413.306553] LustreError: 34650:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [12413.306565] LustreError: 34650:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51b2c9c600/0x8223d6120c4a96db lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d57c3b1e29 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [12413.306575] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [12413.306577] Lustre: Skipped 2 previous similar messages [12429.665011] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 0 seconds [12441.665197] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 22 seconds [12441.786987] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 7 previous similar messages [12454.665425] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 25 seconds [12454.787214] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 5 previous similar messages [12466.665623] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 47 seconds [12466.787419] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 3 previous similar messages [12474.306735] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567184715/real 0] req@ffff8e6926d0c200 x1643302562498000/t0(0) o8->oak-OST0038-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567184726 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12474.634477] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 23 previous similar messages [12479.665822] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 54 seconds [12479.787613] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 1 previous similar message [12492.666025] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 12 seconds [12517.666408] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 4 seconds [12517.787167] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 8 previous similar messages [12547.079063] LustreError: 11-0: oak-OST005b-osc-MDT0000: operation ost_destroy to node 10.0.2.106@o2ib5 failed: rc = -19 [12547.208146] LustreError: Skipped 6 previous similar messages [12547.275864] Lustre: oak-OST005b-osc-MDT0000: Connection to oak-OST005b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [12547.467358] Lustre: Skipped 24 previous similar messages [12553.076951] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567184790/real 0] req@ffff8e69576cec00 x1643302562535632/t0(0) o8->oak-OST0056-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567184801 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12553.404681] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [12555.666972] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 12 seconds [12555.788768] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 11 previous similar messages [12631.668135] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 63 seconds [12631.789931] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 28 previous similar messages [12659.101716] Lustre: oak-MDT0000: haven't heard from client 08f367c8-8a55-4fae-0df1-94540427aae5 (at 10.9.106.70@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966414400, cur 1567184911 expire 1567184761 last 1567184684 [12684.311989] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567184915/real 0] req@ffff8e69cfe4f200 x1643302562589952/t0(0) o8->oak-OST005e-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567184936 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [12684.639723] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 77 previous similar messages [12721.698564] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [12721.861970] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567184673, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6941aab400/0x8223d6120c5f44fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d57e555a1b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [12721.862276] LustreError: 34694:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e69483635c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [12721.862282] LustreError: 34694:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e69483635c0) refcount = 3 [12721.862286] LustreError: 34694:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [12721.862298] LustreError: 34694:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6941aab400/0x8223d6120c5f44fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d57e555a1b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [12769.670325] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 87 seconds [12769.792119] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 71 previous similar messages [12813.096984] Lustre: oak-MDT0000: haven't heard from client 83a34698-98c6-4d7b-6321-3d0a9a56baec (at 10.9.109.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e69673d2000, cur 1567185065 expire 1567184915 last 1567184838 [12925.300840] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [12925.408089] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 4213 previous similar messages [12946.868593] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1567185198/real 1567185198] req@ffff8e68f87e7b00 x1643302562710816/t0(0) o8->oak-OST003d-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567185229 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [12947.209841] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 168 previous similar messages [13032.934346] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [13033.097766] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567184984, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e56b4be2800/0x8223d6120c68cfbe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d580178bb1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [13033.098025] LustreError: 34737:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68f7e2ee40) refcount nonzero (2) after lock cleanup; forcing cleanup. [13033.098027] LustreError: 34737:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68f7e2ee40) refcount = 3 [13033.098029] LustreError: 34737:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [13033.098034] LustreError: 34737:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e56b4be2800/0x8223d6120c68cfbe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d580178bb1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [13033.098039] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [13033.098040] Lustre: Skipped 1 previous similar message [13035.674416] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 353 seconds [13035.797259] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 158 previous similar messages [13343.760179] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [13343.923598] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567185295, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51051e9000/0x8223d6120c730f75 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5801955cf expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [13343.923879] LustreError: 34813:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6914e11b00) refcount nonzero (2) after lock cleanup; forcing cleanup. [13343.923881] LustreError: 34813:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6914e11b00) refcount = 3 [13343.923883] LustreError: 34813:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [13343.923888] LustreError: 34813:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51051e9000/0x8223d6120c730f75 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5801955cf expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [13464.681098] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1567185670/real 1567185716] req@ffff8e69cfe51800 x1643302562913440/t0(0) o8->oak-OST0057-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567185721 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [13465.022349] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 523 previous similar messages [13525.676895] LustreError: 16187:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [13525.784136] LustreError: 16187:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3536 previous similar messages [13552.682433] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 8 seconds [13552.803189] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 335 previous similar messages [13655.037316] LustreError: 34861:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692ab21bc0) refcount = 2 [13655.193436] LustreError: 34861:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [13655.293403] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [13655.401679] Lustre: Skipped 1 previous similar message [13787.120287] Lustre: oak-MDT0000: haven't heard from client bbc5d9b1-7718-6e90-3fc9-06bf9915e197 (at 10.8.18.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e69664c1c00, cur 1567186039 expire 1567185889 last 1567185812 [13863.126831] Lustre: oak-MDT0000: haven't heard from client 2ec7f015-94fd-c74c-014f-77fe4ac1d371 (at 10.9.107.63@o2ib4) in 212 seconds. I think it's dead, and I am evicting it. exp ffff8e696696a800, cur 1567186115 expire 1567185965 last 1567185903 [13939.124348] Lustre: oak-MDT0000: haven't heard from client dcc7eba0-6832-9bcb-3781-1a37ede27eb9 (at 10.8.18.9@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff8e6966bd8800, cur 1567186191 expire 1567186041 last 1567186020 [13939.383435] Lustre: Skipped 1 previous similar message [13962.430714] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [13962.594110] LustreError: Skipped 1 previous similar message [13962.660791] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567185914, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e692f8ef000/0x8223d6120c85b7b3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5801cbc29 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [13962.661022] LustreError: 34914:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68eeefa000) refcount nonzero (2) after lock cleanup; forcing cleanup. [13962.661024] LustreError: 34914:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [13962.661027] LustreError: 34914:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68eeefa000) refcount = 3 [13962.661029] LustreError: 34914:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [13962.661033] LustreError: 34914:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e692f8ef000/0x8223d6120c85b7b3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5801cbc29 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [13962.661034] LustreError: 34914:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [13964.181053] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [14067.662366] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567186264/real 0] req@ffff8e517d3ee000 x1643302563162016/t0(0) o8->oak-OST004c-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567186319 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [14067.990092] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 459 previous similar messages [14126.454940] LustreError: 16602:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [14126.562181] LustreError: 16602:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3265 previous similar messages [14158.691798] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 50 seconds [14158.813597] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 455 previous similar messages [14273.699748] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [14273.808028] Lustre: Skipped 3 previous similar messages [14579.268275] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [14579.431676] LustreError: Skipped 1 previous similar message [14579.498357] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567186531, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e693863b200/0x8223d6120c965035 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5802009bd expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [14579.498602] LustreError: 34989:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692aa960c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [14579.498605] LustreError: 34989:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692aa960c0) refcount = 3 [14579.498606] LustreError: 34989:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [14579.498611] LustreError: 34989:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e693863b200/0x8223d6120c965035 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5802009bd expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [14580.772963] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [14676.699886] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1567186906/real 1567186928] req@ffff8e6935778300 x1643302563433600/t0(0) o8->oak-OST0059-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567186961 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [14677.041141] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 626 previous similar messages [14726.644377] LustreError: 16875:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [14726.751617] LustreError: 16875:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3363 previous similar messages [14764.701128] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.105@o2ib5: 10 seconds [14764.822923] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 637 previous similar messages [14853.131679] Lustre: oak-MDT0000: haven't heard from client 9c476857-28ec-0f31-748c-02430b9ee3fc (at 10.9.108.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e69673d3c00, cur 1567187105 expire 1567186955 last 1567186878 [14888.661315] LustreError: 35023:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e55a52eca80) refcount = 2 [14888.817444] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [14888.925720] Lustre: Skipped 1 previous similar message [14929.188144] Lustre: oak-MDT0000: haven't heard from client 8671c61e-d193-47b2-2724-6c131ec73c0e (at 10.9.106.69@o2ib4) in 170 seconds. I think it's dead, and I am evicting it. exp ffff8e696680b400, cur 1567187181 expire 1567187031 last 1567187011 [14929.449311] Lustre: Skipped 14 previous similar messages [15123.027250] Lustre: oak-MDT0000: Client 7f9dc7aa-e757-45ab-637a-84c853e9a772 (at 10.9.102.7@o2ib4) reconnecting [15123.148009] Lustre: Skipped 1 previous similar message [15161.143509] Lustre: oak-MDT0000: haven't heard from client f9edf48d-1d02-c437-e28b-d6ac5316e92f (at 10.8.7.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966b9bc00, cur 1567187413 expire 1567187263 last 1567187186 [15168.707367] LNetError: 15765:0:(o2iblnd_cb.c:3157:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 8 seconds [15168.827078] LNetError: 15765:0:(o2iblnd_cb.c:3232:kiblnd_check_conns()) Timed out RDMA with 10.0.2.106@o2ib5 (59): c: 0, oc: 0, rc: 8 [15171.681280] Lustre: oak-MDT0000: Client d293f79e-24f2-6bb9-18d4-50d9866885aa (at 10.8.22.27@o2ib6) reconnecting [15195.805768] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [15195.969169] LustreError: Skipped 1 previous similar message [15196.035854] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567187147, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6930e53c00/0x8223d6120ca27c70 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58023b54c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [15196.036153] LustreError: 35074:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68ef7c6900) refcount nonzero (2) after lock cleanup; forcing cleanup. [15196.036157] LustreError: 35074:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [15196.036163] LustreError: 35074:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68ef7c6900) refcount = 3 [15196.036166] LustreError: 35074:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [15196.036187] LustreError: 35074:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6930e53c00/0x8223d6120ca27c70 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58023b54c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [15197.435374] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [15197.564452] sched: RT throttling activated [15207.794099] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.8.28.2@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [15218.070916] Lustre: oak-MDT0000: Client ecee6fd0-c389-3e7b-2625-9e940a76d229 (at 10.9.114.9@o2ib4) reconnecting [15218.191680] Lustre: Skipped 4 previous similar messages [15218.743855] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.9.101.28@o2ib4 (no target). If you are running an HA pair check that the target is mounted on the other server. [15218.951977] LustreError: Skipped 1 previous similar message [15219.708158] LNetError: 15765:0:(o2iblnd_cb.c:3157:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 7 seconds [15219.827876] LNetError: 15765:0:(o2iblnd_cb.c:3232:kiblnd_check_conns()) Timed out RDMA with 10.0.2.105@o2ib5 (58): c: 0, oc: 0, rc: 8 [15231.189614] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.8.23.14@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [15246.036823] LustreError: 167-0: oak-OST0032-osc-MDT0000: This client was evicted by oak-OST0032; in progress operations using this service will fail. [15256.278738] Lustre: oak-MDT0000: Client 002f1e4f-ce11-50c8-8d64-985419ccc517 (at 10.8.23.14@o2ib6) reconnecting [15256.399501] Lustre: Skipped 4 previous similar messages [15271.037419] LustreError: 167-0: oak-OST005c-osc-MDT0000: This client was evicted by oak-OST005c; in progress operations using this service will fail. [15296.037732] LustreError: 167-0: oak-OST0044-osc-MDT0000: This client was evicted by oak-OST0044; in progress operations using this service will fail. [15296.198014] LustreError: Skipped 11 previous similar messages [15327.939664] LustreError: 16875:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [15328.046909] LustreError: 16875:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 6863 previous similar messages [15491.871333] Lustre: 33184:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567187736/real 1567187736] req@ffff8e587373c800 x1643302563810400/t0(0) o104->oak-MDT0000@10.9.110.2@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567187743 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [15492.198020] Lustre: 33184:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 447 previous similar messages [15499.149564] Lustre: oak-MDT0000: haven't heard from client 43a2aa8c-b0d2-3d74-e723-763e98dc0fe3 (at 10.9.110.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966d19400, cur 1567187751 expire 1567187601 last 1567187524 [15499.409692] Lustre: Skipped 21 previous similar messages [15505.761829] LustreError: 35165:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e69427cb2c0) refcount = 2 [15505.917956] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [15506.026229] Lustre: Skipped 87 previous similar messages [15752.193732] Lustre: oak-MDT0000: haven't heard from client 2f251bef-cfad-9797-2623-dc0563708edc (at 10.9.109.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966fcdc00, cur 1567188004 expire 1567187854 last 1567187777 [15752.453862] Lustre: Skipped 6 previous similar messages [15815.366347] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [15815.529775] LustreError: Skipped 1 previous similar message [15815.596479] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567187767, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e692acba800/0x8223d6120cb28104 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5802f547d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [15815.596715] LustreError: 35217:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692c203bc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [15815.596717] LustreError: 35217:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [15815.596719] LustreError: 35217:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692c203bc0) refcount = 3 [15815.596720] LustreError: 35217:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [15815.596725] LustreError: 35217:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e692acba800/0x8223d6120cb28104 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5802f547d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [15816.995990] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [15928.202838] LustreError: 33468:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [15928.310076] LustreError: 33468:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 6151 previous similar messages [16045.148402] Lustre: oak-MDT0000: haven't heard from client 9ca8abd0-3c35-6fb2-f3a0-99a2d5b878c3 (at 10.8.26.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966c77800, cur 1567188297 expire 1567188147 last 1567188070 [16045.408534] Lustre: Skipped 1 previous similar message [16113.100242] Lustre: oak-OST0045-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [16113.215805] Lustre: Skipped 18 previous similar messages [16121.167628] Lustre: oak-MDT0000: haven't heard from client 9cefe124-9e8b-981c-aa16-921fdf8695d5 (at 10.9.113.2@o2ib4) in 211 seconds. I think it's dead, and I am evicting it. exp ffff8e6967392400, cur 1567188373 expire 1567188223 last 1567188162 [16121.427755] Lustre: Skipped 3 previous similar messages [16123.734380] LustreError: 35270:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68f43e80c0) refcount = 2 [16123.890502] LustreError: 35270:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [16304.294953] LustreError: 11-0: oak-OST005b-osc-MDT0000: operation ost_statfs to node 10.0.2.106@o2ib5 failed: rc = -107 [16304.424034] LustreError: Skipped 23 previous similar messages [16304.492793] Lustre: oak-OST005b-osc-MDT0000: Connection to oak-OST005b (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [16304.684274] Lustre: Skipped 21 previous similar messages [16305.332031] LustreError: 11-0: oak-OST0051-osc-MDT0000: operation ost_destroy to node 10.0.2.106@o2ib5 failed: rc = -107 [16305.462153] LustreError: Skipped 2 previous similar messages [16306.502995] Lustre: oak-OST004f-osc-MDT0000: Connection to oak-OST004f (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [16306.694484] Lustre: Skipped 9 previous similar messages [16307.335040] LustreError: 11-0: oak-OST0055-osc-MDT0000: operation ost_statfs to node 10.0.2.106@o2ib5 failed: rc = -107 [16307.464122] LustreError: Skipped 13 previous similar messages [16341.159590] Lustre: oak-MDT0000: haven't heard from client a3820722-3c16-165c-9682-cc2eb1f53cc1 (at 10.8.30.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966d97000, cur 1567188593 expire 1567188443 last 1567188366 [16431.070784] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [16431.234184] LustreError: Skipped 1 previous similar message [16431.300863] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567188382, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e692dfa1400/0x8223d6120cc7a9f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5803c29a6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [16431.301193] LustreError: 35322:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e69444ca240) refcount nonzero (2) after lock cleanup; forcing cleanup. [16431.301195] LustreError: 35322:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [16431.301197] LustreError: 35322:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e69444ca240) refcount = 3 [16431.301199] LustreError: 35322:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [16431.301204] LustreError: 35322:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e692dfa1400/0x8223d6120cc7a9f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5803c29a6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [16431.301205] LustreError: 35322:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [16432.821167] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [16439.300914] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1567188675/real 0] req@ffff8e68cae8aa00 x1643302564328544/t0(0) o8->oak-OST0045-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 520/544 e 0 to 1 dl 1567188691 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [16439.628643] Lustre: 15776:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [16528.783587] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [16528.890838] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 23368 previous similar messages [16740.088998] LustreError: 35364:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e69162f5e00) refcount = 2 [16740.245125] LustreError: 35364:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [16740.345086] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [16740.453358] Lustre: Skipped 16 previous similar messages [16772.732064] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Timed out tx for 10.0.2.106@o2ib5: 2 seconds [16772.852815] LNet: 15765:0:(o2iblnd_cb.c:3202:kiblnd_check_conns()) Skipped 264 previous similar messages [16915.091425] LustreError: 167-0: oak-OST0033-osc-MDT0000: This client was evicted by oak-OST0033; in progress operations using this service will fail. [16915.251709] LustreError: Skipped 10 previous similar messages [16940.091938] LustreError: 167-0: oak-OST0045-osc-MDT0000: This client was evicted by oak-OST0045; in progress operations using this service will fail. [16940.252222] LustreError: Skipped 2 previous similar messages [16965.092155] LustreError: 167-0: oak-OST003f-osc-MDT0000: This client was evicted by oak-OST003f; in progress operations using this service will fail. [16965.252446] LustreError: Skipped 13 previous similar messages [17046.304273] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [17046.467679] LustreError: Skipped 1 previous similar message [17046.534363] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567188998, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6941fec600/0x8223d6120cf3d730 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5804dd23a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [17046.534509] LustreError: 35442:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6926c05740) refcount nonzero (2) after lock cleanup; forcing cleanup. [17046.534511] LustreError: 35442:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [17046.534513] LustreError: 35442:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6926c05740) refcount = 3 [17046.534515] LustreError: 35442:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [17046.534520] LustreError: 35442:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6941fec600/0x8223d6120cf3d730 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5804dd23a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [17046.534521] LustreError: 35442:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [17048.054644] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [17128.800116] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [17128.907362] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 96910 previous similar messages [17134.422498] Lustre: DEBUG MARKER: Fri Aug 30 11:23:06 2019 [17187.205562] Lustre: oak-MDT0000: haven't heard from client 6dbbef9c-ba17-024b-445e-6852e6e40407 (at 10.9.113.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966c71800, cur 1567189439 expire 1567189289 last 1567189212 [17187.466738] Lustre: Skipped 3 previous similar messages [17194.906582] Lustre: 33152:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567189439/real 1567189439] req@ffff8e4f3fe49500 x1643302564764048/t0(0) o104->oak-MDT0000@10.9.104.27@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567189446 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [17195.234319] Lustre: 33152:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 196 previous similar messages [17263.175594] Lustre: oak-MDT0000: haven't heard from client a3f431ac-6a11-d105-7874-e1e327053b16 (at 10.9.104.25@o2ib4) in 171 seconds. I think it's dead, and I am evicting it. exp ffff8e696680b800, cur 1567189515 expire 1567189365 last 1567189344 [17263.436858] LustreError: 33152:0:(client.c:1166:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8e4f40227500 x1643302564817120/t0(0) o104->oak-MDT0000@10.9.104.27@o2ib4:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [17354.298305] LustreError: 35513:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e68b52d8780) refcount = 2 [17354.454431] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [17354.562716] Lustre: Skipped 70 previous similar messages [17570.170799] Lustre: oak-MDT0000: haven't heard from client 00180c61-1704-c180-93af-767f42e9a309 (at 10.9.110.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966e0bc00, cur 1567189822 expire 1567189672 last 1567189595 [17570.431964] Lustre: Skipped 1 previous similar message [17659.996784] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [17660.160286] LustreError: Skipped 1 previous similar message [17660.227005] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567189611, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6959820600/0x8223d6120e0d5535 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d584ed50c6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [17660.227170] LustreError: 35558:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68fffe0600) refcount nonzero (2) after lock cleanup; forcing cleanup. [17660.227172] LustreError: 35558:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [17660.227175] LustreError: 35558:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68fffe0600) refcount = 3 [17660.227176] LustreError: 35558:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [17660.227181] LustreError: 35558:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6959820600/0x8223d6120e0d5535 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d584ed50c6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [17661.626569] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [17728.811774] LustreError: 34429:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [17728.919012] LustreError: 34429:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 235000 previous similar messages [17958.680623] Lustre: oak-MDT0000: Connection restored to 2b79fdf0-f80f-9a30-2ccb-9637f412a74e (at 10.8.15.4@o2ib6) [17958.803474] Lustre: Skipped 21 previous similar messages [17967.296771] LustreError: 35593:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e51a14bf200) refcount = 2 [18273.505308] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [18273.668708] LustreError: Skipped 1 previous similar message [18273.735391] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567190225, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e68fbb49600/0x8223d6120f1f1cae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d585d179f3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [18273.735520] LustreError: 35644:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68f7784000) refcount nonzero (2) after lock cleanup; forcing cleanup. [18273.735522] LustreError: 35644:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [18273.735525] LustreError: 35644:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68f7784000) refcount = 3 [18273.735527] LustreError: 35644:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [18273.735533] LustreError: 35644:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e68fbb49600/0x8223d6120f1f1cae lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d585d179f3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [18275.134940] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [18328.825289] LustreError: 33466:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [18328.932530] LustreError: 33466:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 240450 previous similar messages [18582.437337] LustreError: 35692:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6831716a80) refcount = 2 [18582.593502] LustreError: 35692:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [18582.693477] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [18582.801751] Lustre: Skipped 11 previous similar messages [18891.077884] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [18891.241289] LustreError: Skipped 1 previous similar message [18891.307972] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567190842, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e68c9fb3800/0x8223d6120feeeb5e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d586727e45 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [18891.308230] LustreError: 35743:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6810aa0540) refcount nonzero (2) after lock cleanup; forcing cleanup. [18891.308232] LustreError: 35743:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [18891.308234] LustreError: 35743:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6810aa0540) refcount = 3 [18891.308236] LustreError: 35743:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [18891.308240] LustreError: 35743:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e68c9fb3800/0x8223d6120feeeb5e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d586727e45 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [18891.308241] LustreError: 35743:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [18892.828264] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [18928.855628] LustreError: 16876:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [18928.962873] LustreError: 16876:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 178856 previous similar messages [19202.601897] LustreError: 35784:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e577a038cc0) refcount = 2 [19202.764272] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [19202.872570] Lustre: Skipped 2 previous similar messages [19509.868478] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [19510.031778] LustreError: Skipped 1 previous similar message [19510.098465] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567191461, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e69376b6000/0x8223d612107acc40 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d588e5dcd2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [19510.098646] LustreError: 35849:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6806390480) refcount nonzero (2) after lock cleanup; forcing cleanup. [19510.098648] LustreError: 35849:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [19510.098650] LustreError: 35849:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6806390480) refcount = 3 [19510.098652] LustreError: 35849:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [19510.098656] LustreError: 35849:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e69376b6000/0x8223d612107acc40 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d588e5dcd2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [19511.497979] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [19528.868141] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [19528.975385] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 359078 previous similar messages [19818.302486] LustreError: 35894:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e67f82d8000) refcount = 2 [19818.458608] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [19818.566883] Lustre: Skipped 2 previous similar messages [20125.842960] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [20126.006362] LustreError: Skipped 1 previous similar message [20126.073063] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567192077, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5731ec3000/0x8223d61212a67b89 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5898ae624 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [20126.073204] LustreError: 35942:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e67f0353980) refcount nonzero (2) after lock cleanup; forcing cleanup. [20126.073205] LustreError: 35942:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [20126.073208] LustreError: 35942:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67f0353980) refcount = 3 [20126.073209] LustreError: 35942:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [20126.073213] LustreError: 35942:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5731ec3000/0x8223d61212a67b89 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5898ae624 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [20127.472619] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [20128.880842] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [20128.988076] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 635264 previous similar messages [20434.900114] LustreError: 35978:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67e771b2c0) refcount = 2 [20435.056233] LustreError: 35978:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [20435.156198] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [20435.264489] Lustre: Skipped 2 previous similar messages [20728.900217] LustreError: 34440:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [20729.007456] LustreError: 34440:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 630381 previous similar messages [20741.772520] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [20741.935923] LustreError: Skipped 1 previous similar message [20742.002611] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567192693, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e693daf3c00/0x8223d61214c34f8d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58b82ef77 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [20742.002795] LustreError: 36052:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e67e32946c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [20742.002797] LustreError: 36052:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [20742.002799] LustreError: 36052:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67e32946c0) refcount = 3 [20742.002800] LustreError: 36052:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [20742.002805] LustreError: 36052:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e693daf3c00/0x8223d61214c34f8d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58b82ef77 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [20742.002806] LustreError: 36052:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [20743.522920] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [20871.226299] Lustre: oak-MDT0000: haven't heard from client f63f27eb-bc6e-176e-00ae-1840a6228e8e (at 10.9.103.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966847400, cur 1567193123 expire 1567192973 last 1567192896 [21043.225301] Lustre: oak-MDT0000: haven't heard from client 5a341d41-a090-26f7-4884-7fab2fd288b7 (at 10.9.104.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966c3f000, cur 1567193295 expire 1567193145 last 1567193068 [21043.486466] Lustre: Skipped 1 previous similar message [21050.548648] LustreError: 36093:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67d86b3800) refcount = 2 [21050.704777] LustreError: 36093:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [21050.804769] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [21050.913041] Lustre: Skipped 1 previous similar message [21239.227263] Lustre: oak-MDT0000: haven't heard from client 592fd845-3cea-a0bc-c25c-d66b845b1e81 (at 10.9.110.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966e7a400, cur 1567193491 expire 1567193341 last 1567193264 [21328.913752] LustreError: 34446:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [21329.021001] LustreError: 34446:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 651518 previous similar messages [21358.052067] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [21358.215471] LustreError: Skipped 1 previous similar message [21358.282154] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567193309, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e53b38e0600/0x8223d61215eba7e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58c205e0c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [21358.282355] LustreError: 36146:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e67dc353500) refcount nonzero (2) after lock cleanup; forcing cleanup. [21358.282357] LustreError: 36146:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [21358.282359] LustreError: 36146:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67dc353500) refcount = 3 [21358.282361] LustreError: 36146:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [21358.282365] LustreError: 36146:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e53b38e0600/0x8223d61215eba7e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58c205e0c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [21358.282366] LustreError: 36146:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [21359.802429] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [21669.908205] LustreError: 36180:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6913ea2480) refcount = 2 [21670.064331] LustreError: 36180:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [21670.164323] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [21670.272651] Lustre: Skipped 3 previous similar messages [21928.924292] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [21929.031533] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 399054 previous similar messages [21977.482749] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [21977.646157] LustreError: Skipped 1 previous similar message [21977.712865] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567193929, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e690baaa800/0x8223d6121711452c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58c673f2b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [21977.713093] LustreError: 36221:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68fab33200) refcount nonzero (2) after lock cleanup; forcing cleanup. [21977.713097] LustreError: 36221:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [21977.713103] LustreError: 36221:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68fab33200) refcount = 3 [21977.713107] LustreError: 36221:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [21977.713119] LustreError: 36221:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e690baaa800/0x8223d6121711452c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58c673f2b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [21977.713122] LustreError: 36221:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [21979.233166] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [22288.410861] LustreError: 36265:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e68497049c0) refcount = 2 [22288.567012] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [22288.675289] Lustre: Skipped 1 previous similar message [22528.960008] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [22529.067247] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 296273 previous similar messages [22598.263322] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [22598.426725] LustreError: Skipped 1 previous similar message [22598.493418] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567194550, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e68ef65fe00/0x8223d61217e6e523 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58cc46839 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [22598.493664] LustreError: 36310:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e683d2a2a80) refcount nonzero (2) after lock cleanup; forcing cleanup. [22598.493666] LustreError: 36310:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [22598.493668] LustreError: 36310:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e683d2a2a80) refcount = 3 [22598.493670] LustreError: 36310:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [22598.493674] LustreError: 36310:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e68ef65fe00/0x8223d61217e6e523 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58cc46839 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [22599.892972] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [22909.159537] LustreError: 36354:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e690f737680) refcount = 2 [22909.315661] LustreError: 36354:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [22909.415646] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [22909.523923] Lustre: Skipped 1 previous similar message [23128.971881] LustreError: 34446:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [23129.079131] LustreError: 34446:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 356866 previous similar messages [23214.522885] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [23214.686290] LustreError: Skipped 1 previous similar message [23214.752978] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567195166, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6937648800/0x8223d61218bf3892 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58cfd9b36 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [23214.753191] LustreError: 36418:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e67ba606000) refcount nonzero (2) after lock cleanup; forcing cleanup. [23214.753193] LustreError: 36418:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [23214.753195] LustreError: 36418:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67ba606000) refcount = 3 [23214.753196] LustreError: 36418:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [23214.753200] LustreError: 36418:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6937648800/0x8223d61218bf3892 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58cfd9b36 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [23214.753202] LustreError: 36418:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [23216.273271] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [23282.259192] Lustre: oak-MDT0000: haven't heard from client 823afa50-83a6-c99a-fa8e-0f146acce0e2 (at 10.8.21.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966d19800, cur 1567195534 expire 1567195384 last 1567195307 [23522.632836] LustreError: 36455:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67b2605bc0) refcount = 2 [23522.788955] LustreError: 36455:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [23522.888916] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [23522.997196] Lustre: Skipped 5 previous similar messages [23728.981882] LustreError: 34443:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [23729.089124] LustreError: 34443:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 397688 previous similar messages [23832.989574] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [23833.152974] LustreError: Skipped 1 previous similar message [23833.219657] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567195784, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6909a06c00/0x8223d61219892538 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58d152eda expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [23833.219830] LustreError: 36508:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e69237c95c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [23833.219834] LustreError: 36508:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [23833.219840] LustreError: 36508:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e69237c95c0) refcount = 3 [23833.219843] LustreError: 36508:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [23833.219856] LustreError: 36508:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6909a06c00/0x8223d61219892538 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58d152eda expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [23833.219861] LustreError: 36508:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [23834.739975] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [24144.169679] LustreError: 36544:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68f0e07c80) refcount = 2 [24144.325804] LustreError: 36544:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [24144.425785] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [24144.534060] Lustre: Skipped 2 previous similar messages [24328.994759] LustreError: 34435:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [24329.101997] LustreError: 34435:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 450650 previous similar messages [24449.695160] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [24449.858563] LustreError: Skipped 1 previous similar message [24449.925260] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567196401, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e693decc200/0x8223d6121a60965b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58d267f5b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [24449.925500] LustreError: 36589:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e67eb500fc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [24449.925501] LustreError: 36589:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [24449.925504] LustreError: 36589:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67eb500fc0) refcount = 3 [24449.925505] LustreError: 36589:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [24449.925510] LustreError: 36589:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e693decc200/0x8223d6121a60965b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58d267f5b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [24449.925511] LustreError: 36589:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [24451.445560] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [24756.775213] LustreError: 36640:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e6902be3e00) refcount = 2 [24756.931346] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [24757.039625] Lustre: Skipped 1 previous similar message [24929.005641] LustreError: 16854:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [24929.112883] LustreError: 16854:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 788388 previous similar messages [25062.970685] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [25063.134089] LustreError: Skipped 1 previous similar message [25063.200775] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567197014, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e680539f000/0x8223d6121baf41d1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58d40b91f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [25063.201025] LustreError: 36679:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e67e4444300) refcount nonzero (2) after lock cleanup; forcing cleanup. [25063.201027] LustreError: 36679:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [25063.201029] LustreError: 36679:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67e4444300) refcount = 3 [25063.201031] LustreError: 36679:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [25063.201035] LustreError: 36679:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e680539f000/0x8223d6121baf41d1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58d40b91f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [25064.600339] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [25369.821721] LustreError: 36720:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67e0f13500) refcount = 2 [25369.977845] LustreError: 36720:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [25370.077815] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [25370.186122] Lustre: Skipped 2 previous similar messages [25529.015890] LustreError: 34422:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [25529.123125] LustreError: 34422:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1294611 previous similar messages [25677.319228] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [25677.482636] LustreError: Skipped 1 previous similar message [25677.549318] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567197629, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e680764fa00/0x8223d6121de650b1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58d54c0bb expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [25677.549560] LustreError: 36764:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68f3b466c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [25677.549562] LustreError: 36764:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [25677.549564] LustreError: 36764:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68f3b466c0) refcount = 3 [25677.549566] LustreError: 36764:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [25677.549570] LustreError: 36764:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e680764fa00/0x8223d6121de650b1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58d54c0bb expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [25677.549571] LustreError: 36764:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [25679.069630] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [25721.297667] Lustre: oak-MDT0000: haven't heard from client ebad0411-7871-9d40-80cc-2a88b792749f (at 10.8.8.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966efec00, cur 1567197973 expire 1567197823 last 1567197746 [25988.140514] LustreError: 36810:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67eaad3b00) refcount = 2 [25988.296634] LustreError: 36810:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [25988.396599] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [25988.504906] Lustre: Skipped 1 previous similar message [26129.027952] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [26129.135202] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 826583 previous similar messages [26295.017018] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [26295.180438] LustreError: Skipped 1 previous similar message [26295.247126] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567198246, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e57a27c7600/0x8223d6121fd11f7e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58d6e3a53 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [26295.247366] LustreError: 36856:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e678e3a0a80) refcount nonzero (2) after lock cleanup; forcing cleanup. [26295.247367] LustreError: 36856:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [26295.247369] LustreError: 36856:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e678e3a0a80) refcount = 3 [26295.247371] LustreError: 36856:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [26295.247376] LustreError: 36856:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e57a27c7600/0x8223d6121fd11f7e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58d6e3a53 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [26295.247377] LustreError: 36856:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [26296.767450] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [26605.588054] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [26605.696337] Lustre: Skipped 2 previous similar messages [26729.046766] LustreError: 34444:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [26729.154020] LustreError: 34444:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 727247 previous similar messages [26913.657672] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [26913.821086] LustreError: Skipped 1 previous similar message [26913.887785] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567198865, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e519ff94600/0x8223d6122152fac3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58d8e7497 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [26913.888026] LustreError: 36961:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e677cf72000) refcount nonzero (2) after lock cleanup; forcing cleanup. [26913.888028] LustreError: 36961:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e677cf72000) refcount = 3 [26913.888030] LustreError: 36961:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [26913.888034] LustreError: 36961:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e519ff94600/0x8223d6122152fac3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58d8e7497 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [26915.162428] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [27224.685773] LustreError: 37016:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e6769adf2c0) refcount = 2 [27224.841909] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [27224.950194] Lustre: Skipped 1 previous similar message [27329.058208] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [27329.165452] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 751701 previous similar messages [27533.506284] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [27533.669685] LustreError: Skipped 1 previous similar message [27533.736365] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567199485, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e518e3ef400/0x8223d61222f8ceea lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58dad1dd5 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [27533.736571] LustreError: 37061:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e677aa8b680) refcount nonzero (2) after lock cleanup; forcing cleanup. [27533.736573] LustreError: 37061:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [27533.736575] LustreError: 37061:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e677aa8b680) refcount = 3 [27533.736576] LustreError: 37061:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [27533.736580] LustreError: 37061:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e518e3ef400/0x8223d61222f8ceea lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58dad1dd5 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [27535.135907] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [27841.320498] LustreError: 37103:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6940a7fbc0) refcount = 2 [27841.476623] LustreError: 37103:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [27841.576623] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [27841.684902] Lustre: Skipped 2 previous similar messages [27929.069615] LustreError: 16929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [27929.176855] LustreError: 16929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 676806 previous similar messages [28149.591937] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [28149.755349] LustreError: Skipped 1 previous similar message [28149.822041] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567200101, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51ed676200/0x8223d612246b6653 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58dca01a2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [28149.822309] LustreError: 37143:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68cfbdcf00) refcount nonzero (2) after lock cleanup; forcing cleanup. [28149.822310] LustreError: 37143:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [28149.822312] LustreError: 37143:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68cfbdcf00) refcount = 3 [28149.822314] LustreError: 37143:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [28149.822319] LustreError: 37143:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51ed676200/0x8223d612246b6653 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58dca01a2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [28149.822320] LustreError: 37143:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [28151.342352] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [28460.987059] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [28461.095340] Lustre: Skipped 1 previous similar message [28526.339894] Lustre: oak-MDT0000: haven't heard from client 7a722bd6-422b-ea50-5a1b-9cb9b66e4b7c (at 10.8.27.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e57088b5000, cur 1567200778 expire 1567200628 last 1567200551 [28529.083664] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [28529.190907] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 737774 previous similar messages [28770.880606] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [28771.044014] LustreError: Skipped 1 previous similar message [28771.110700] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567200722, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e67d9a21e00/0x8223d61225f190c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58ddfe0d0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [28771.110872] LustreError: 37251:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e693df40840) refcount nonzero (2) after lock cleanup; forcing cleanup. [28771.110874] LustreError: 37251:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e693df40840) refcount = 3 [28771.110876] LustreError: 37251:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [28771.110880] LustreError: 37251:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e67d9a21e00/0x8223d61225f190c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58ddfe0d0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [28772.385368] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [29078.920660] LustreError: 37301:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e693ed99e00) refcount = 2 [29079.083098] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [29079.191384] Lustre: Skipped 2 previous similar messages [29129.102284] LustreError: 34422:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [29129.209530] LustreError: 34422:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 505337 previous similar messages [29387.966215] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [29388.129632] LustreError: Skipped 1 previous similar message [29388.196322] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567201339, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e693965ac00/0x8223d61228cb2585 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58df650c0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [29388.196574] LustreError: 37348:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692ab9ce40) refcount nonzero (2) after lock cleanup; forcing cleanup. [29388.196578] LustreError: 37348:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [29388.196584] LustreError: 37348:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692ab9ce40) refcount = 3 [29388.196588] LustreError: 37348:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [29388.196600] LustreError: 37348:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e693965ac00/0x8223d61228cb2585 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58df650c0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [29389.595879] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [29699.536457] LustreError: 37399:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e68f42be540) refcount = 2 [29699.692624] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [29699.800906] Lustre: Skipped 1 previous similar message [29729.117006] LustreError: 16854:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [29729.224246] LustreError: 16854:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 310960 previous similar messages [29903.362395] Lustre: oak-MDT0000: haven't heard from client 065e390d-53d4-bedc-4d13-b615778a3163 (at 10.8.8.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966dcf400, cur 1567202155 expire 1567202005 last 1567201928 [30008.929908] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [30009.093315] LustreError: Skipped 1 previous similar message [30009.159999] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567201960, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e577b3a2000/0x8223d6122c92e3b0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58ec8f91b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [30009.160255] LustreError: 37443:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6868a72900) refcount nonzero (2) after lock cleanup; forcing cleanup. [30009.160257] LustreError: 37443:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [30009.160259] LustreError: 37443:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6868a72900) refcount = 3 [30009.160260] LustreError: 37443:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [30009.160265] LustreError: 37443:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e577b3a2000/0x8223d6122c92e3b0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58ec8f91b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [30010.559526] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [30317.267180] LustreError: 37508:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67566da180) refcount = 1 [30317.423311] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [30317.531584] Lustre: Skipped 1 previous similar message [30329.168489] LustreError: 16874:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [30329.275730] LustreError: 16874:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 361297 previous similar messages [30626.909596] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [30627.073004] LustreError: Skipped 1 previous similar message [30627.139691] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567202578, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6924ed2600/0x8223d6123113bae8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58ef369f4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [30627.139933] LustreError: 37557:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e66ef6abe00) refcount nonzero (2) after lock cleanup; forcing cleanup. [30627.139934] LustreError: 37557:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [30627.139937] LustreError: 37557:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e66ef6abe00) refcount = 3 [30627.139938] LustreError: 37557:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [30627.139943] LustreError: 37557:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6924ed2600/0x8223d6123113bae8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58ef369f4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [30628.539243] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [30929.194893] LustreError: 16865:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [30929.302140] LustreError: 16865:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 205171 previous similar messages [30937.373816] LustreError: 37609:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e66ef2bb8c0) refcount = 2 [30937.529935] LustreError: 37609:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [30937.629901] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [30937.738180] Lustre: Skipped 1 previous similar message [31246.339223] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [31246.502626] LustreError: Skipped 1 previous similar message [31246.569323] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567203197, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51aefbb400/0x8223d6123262f0cf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d58fad54b4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [31246.569531] LustreError: 37649:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e67b2682e40) refcount nonzero (2) after lock cleanup; forcing cleanup. [31246.569533] LustreError: 37649:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [31246.569535] LustreError: 37649:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e67b2682e40) refcount = 3 [31246.569537] LustreError: 37649:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [31246.569541] LustreError: 37649:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51aefbb400/0x8223d6123262f0cf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d58fad54b4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [31246.569542] LustreError: 37649:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [31248.089683] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [31529.205714] LustreError: 33469:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [31529.312951] LustreError: 33469:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 285170 previous similar messages [31553.286317] LustreError: 37701:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6941fb8600) refcount = 2 [31553.442438] LustreError: 37701:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [31553.542409] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [31553.650715] Lustre: Skipped 1 previous similar message [31860.300779] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [31860.464187] LustreError: Skipped 1 previous similar message [31860.530871] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567203811, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e67d67e4600/0x8223d612334968c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5906f5a14 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [31860.531032] LustreError: 37735:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e676e3d5b00) refcount nonzero (2) after lock cleanup; forcing cleanup. [31860.531044] LustreError: 37735:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [31860.531047] LustreError: 37735:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e676e3d5b00) refcount = 3 [31860.531048] LustreError: 37735:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [31860.531053] LustreError: 37735:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e67d67e4600/0x8223d612334968c6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5906f5a14 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [31860.531054] LustreError: 37735:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [31862.051172] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [31901.395357] Lustre: oak-MDT0000: haven't heard from client b34908ca-72df-f8f4-e62e-1c4e0ea09ce4 (at 10.8.17.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966572400, cur 1567204153 expire 1567204003 last 1567203926 [32129.226381] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [32129.333622] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 208207 previous similar messages [32168.526883] LustreError: 37786:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e693e26c3c0) refcount = 2 [32168.683006] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [32168.791280] Lustre: Skipped 2 previous similar messages [32475.993329] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [32476.156736] LustreError: Skipped 1 previous similar message [32476.223420] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567204427, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e693b3a3c00/0x8223d61233cbe12a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d591231c1b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [32476.223651] LustreError: 37827:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6940c79500) refcount nonzero (2) after lock cleanup; forcing cleanup. [32476.223652] LustreError: 37827:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [32476.223655] LustreError: 37827:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6940c79500) refcount = 3 [32476.223656] LustreError: 37827:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [32476.223661] LustreError: 37827:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e693b3a3c00/0x8223d61233cbe12a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d591231c1b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [32477.622965] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [32729.244358] LustreError: 34423:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [32729.351599] LustreError: 34423:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 140881 previous similar messages [32784.208395] LustreError: 37880:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e5189dfb5c0) refcount = 2 [32784.370761] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [32784.479038] Lustre: Skipped 4 previous similar messages [32920.421205] Lustre: oak-MDT0000: haven't heard from client 89b8a900-beb2-ea29-4743-82448155ea4a (at 10.9.104.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966e78c00, cur 1567205172 expire 1567205022 last 1567204945 [33082.413656] Lustre: oak-MDT0000: haven't heard from client f15daeaa-8e65-a2d3-c7a7-8868ac023d74 (at 10.9.104.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966fccc00, cur 1567205334 expire 1567205184 last 1567205107 [33092.370935] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [33092.534343] LustreError: Skipped 1 previous similar message [33092.601031] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567205043, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e692f96e000/0x8223d612344bb1df lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d591d9f2ac expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [33092.601191] LustreError: 37945:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692907ec00) refcount nonzero (2) after lock cleanup; forcing cleanup. [33092.601192] LustreError: 37945:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [33092.601195] LustreError: 37945:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692907ec00) refcount = 3 [33092.601196] LustreError: 37945:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [33092.601201] LustreError: 37945:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e692f96e000/0x8223d612344bb1df lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d591d9f2ac expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [33094.000591] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [33329.266037] LustreError: 34461:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [33329.373431] LustreError: 34461:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 167896 previous similar messages [33402.348114] LustreError: 37999:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e52b948f380) refcount = 2 [33402.504241] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [33402.612516] Lustre: Skipped 6 previous similar messages [33651.421261] Lustre: oak-MDT0000: haven't heard from client b4eab8b1-5659-0d20-f211-463eff4520a7 (at 10.9.104.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966e45000, cur 1567205903 expire 1567205753 last 1567205676 [33710.911649] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [33711.075048] LustreError: Skipped 1 previous similar message [33711.141729] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567205662, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4e45e12c00/0x8223d61237e873ee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5929a83a7 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [33711.141984] LustreError: 38035:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e52b975f980) refcount nonzero (2) after lock cleanup; forcing cleanup. [33711.141986] LustreError: 38035:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [33711.141988] LustreError: 38035:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e52b975f980) refcount = 3 [33711.141990] LustreError: 38035:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [33711.141994] LustreError: 38035:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4e45e12c00/0x8223d61237e873ee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5929a83a7 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [33712.541261] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [33929.280654] LustreError: 34431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [33929.387901] LustreError: 34431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 214011 previous similar messages [34022.031755] LustreError: 38107:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5274fa83c0) refcount = 2 [34022.187883] LustreError: 38107:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [34022.287854] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [34022.396129] Lustre: Skipped 3 previous similar messages [34332.039384] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [34332.202786] LustreError: Skipped 1 previous similar message [34332.269470] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567206283, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d2f382e00/0x8223d61239064463 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5936fa527 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [34332.269796] LustreError: 38136:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56b2428780) refcount nonzero (2) after lock cleanup; forcing cleanup. [34332.269798] LustreError: 38136:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [34332.269801] LustreError: 38136:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56b2428780) refcount = 3 [34332.269802] LustreError: 38136:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [34332.269807] LustreError: 38136:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d2f382e00/0x8223d61239064463 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5936fa527 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [34332.269808] LustreError: 38136:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [34333.789786] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [34529.291632] LustreError: 16184:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [34529.398887] LustreError: 16184:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 153979 previous similar messages [34643.311653] LustreError: 38190:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5448430780) refcount = 2 [34643.467780] LustreError: 38190:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [34643.567745] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [34643.676023] Lustre: Skipped 2 previous similar messages [34884.440768] Lustre: oak-MDT0000: haven't heard from client 004ce3c6-0436-adc7-8bb1-1ac9870e972e (at 10.9.113.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e53d266e800, cur 1567207136 expire 1567206986 last 1567206909 [34953.096169] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [34953.259573] LustreError: Skipped 1 previous similar message [34953.326259] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567206904, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e58c7c86a00/0x8223d61239ad27a1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d594207dc3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [34953.326492] LustreError: 38245:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e57cf1a03c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [34953.326494] LustreError: 38245:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [34953.326497] LustreError: 38245:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e57cf1a03c0) refcount = 3 [34953.326498] LustreError: 38245:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [34953.326502] LustreError: 38245:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e58c7c86a00/0x8223d61239ad27a1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d594207dc3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [34953.326504] LustreError: 38245:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [34954.846539] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [35129.306886] LustreError: 16874:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [35129.414125] LustreError: 16874:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 158298 previous similar messages [35261.169318] LustreError: 38289:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e580efdc780) refcount = 2 [35261.325440] LustreError: 38289:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [35261.425413] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [35261.533692] Lustre: Skipped 1 previous similar message [35283.450781] Lustre: oak-MDT0000: haven't heard from client 95982b57-8148-6325-391f-06aa00b62b34 (at 10.9.113.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e690a77ec00, cur 1567207535 expire 1567207385 last 1567207308 [35567.868754] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [35568.032156] LustreError: Skipped 1 previous similar message [35568.098849] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567207519, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e574ea84000/0x8223d6123a584c46 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d595081b3c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [35568.099049] LustreError: 38334:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5797420e40) refcount nonzero (2) after lock cleanup; forcing cleanup. [35568.099050] LustreError: 38334:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [35568.099053] LustreError: 38334:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5797420e40) refcount = 3 [35568.099054] LustreError: 38334:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [35568.099059] LustreError: 38334:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e574ea84000/0x8223d6123a584c46 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d595081b3c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [35568.099060] LustreError: 38334:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [35569.619381] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [35729.325427] LustreError: 34456:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [35729.432697] LustreError: 34456:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 304155 previous similar messages [35846.466505] Lustre: oak-MDT0000: haven't heard from client b2b3150d-9ecd-5306-ec97-2b2f41623c7d (at 10.9.109.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6940d33000, cur 1567208098 expire 1567207948 last 1567207871 [35879.772925] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [35879.881224] Lustre: Skipped 2 previous similar messages [35922.456635] Lustre: oak-MDT0000: haven't heard from client 97e72cf1-cb10-36c8-6ca9-09b825ae4da6 (at 10.8.28.4@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff8e6966f5dc00, cur 1567208174 expire 1567208024 last 1567207952 [36181.068897] perf: interrupt took too long (2519 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [36192.211531] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [36192.374976] LustreError: Skipped 1 previous similar message [36192.441692] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567208143, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e690c707e00/0x8223d61248991a7e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d595f57247 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [36192.441903] LustreError: 38433:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e695d36e000) refcount nonzero (2) after lock cleanup; forcing cleanup. [36192.441905] LustreError: 38433:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e695d36e000) refcount = 3 [36192.441907] LustreError: 38433:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [36192.441912] LustreError: 38433:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e690c707e00/0x8223d61248991a7e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d595f57247 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [36193.716353] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [36329.336675] LustreError: 16604:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [36329.443925] LustreError: 16604:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 290665 previous similar messages [36500.114560] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [36500.222854] Lustre: Skipped 1 previous similar message [36806.001137] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [36806.164554] LustreError: Skipped 1 previous similar message [36806.231251] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567208757, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e67a47e4600/0x8223d61259cca31a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d596ff84ed expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [36806.231457] LustreError: 38539:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e603dac8900) refcount nonzero (2) after lock cleanup; forcing cleanup. [36806.231459] LustreError: 38539:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e603dac8900) refcount = 3 [36806.231461] LustreError: 38539:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [36806.231465] LustreError: 38539:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e67a47e4600/0x8223d61259cca31a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d596ff84ed expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [36807.505895] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [36929.348549] LustreError: 37927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [36929.455817] LustreError: 37927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 207141 previous similar messages [37114.694197] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [37114.802501] Lustre: Skipped 3 previous similar messages [37424.038817] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [37424.202233] LustreError: Skipped 1 previous similar message [37424.268936] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567209375, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6802ef7800/0x8223d61268ace42e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d597f84686 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [37424.269120] LustreError: 38645:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5d64e1aa80) refcount nonzero (2) after lock cleanup; forcing cleanup. [37424.269123] LustreError: 38645:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5d64e1aa80) refcount = 3 [37424.269124] LustreError: 38645:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [37424.269129] LustreError: 38645:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6802ef7800/0x8223d61268ace42e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d597f84686 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [37425.543566] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [37529.364269] LustreError: 16861:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [37529.471567] LustreError: 16861:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 164310 previous similar messages [37731.937805] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [37732.046101] Lustre: Skipped 1 previous similar message [38040.347520] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [38040.511034] LustreError: Skipped 1 previous similar message [38040.577994] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567209991, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5bcbf49600/0x8223d61278124e15 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d598b0ef15 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [38040.578125] LustreError: 38745:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e644fcd6000) refcount nonzero (2) after lock cleanup; forcing cleanup. [38040.578127] LustreError: 38745:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e644fcd6000) refcount = 3 [38040.578129] LustreError: 38745:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [38040.578134] LustreError: 38745:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5bcbf49600/0x8223d61278124e15 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d598b0ef15 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [38041.852611] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [38129.406168] LustreError: 34437:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [38129.513407] LustreError: 34437:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 104268 previous similar messages [38348.328637] LustreError: 38783:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e690feff500) refcount = 2 [38348.484779] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [38348.593062] Lustre: Skipped 3 previous similar messages [38393.865091] Lustre: oak-MDT0000: Client 03eecfa2-5c8c-511b-9ceb-f6884250addf (at 10.8.10.10@o2ib6) reconnecting [38393.985850] Lustre: Skipped 2 previous similar messages [38445.308109] Lustre: oak-MDT0000: Client 04c9f1ee-f3ed-b3be-e062-2beb2d764912 (at 10.8.11.5@o2ib6) reconnecting [38655.676266] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [38655.839672] LustreError: Skipped 1 previous similar message [38655.906385] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567210607, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5fd42bae00/0x8223d61297ae1003 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5998e2784 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [38655.911328] LustreError: 38828:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e60c9b7db00) refcount nonzero (2) after lock cleanup; forcing cleanup. [38655.911330] LustreError: 38828:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [38655.911333] LustreError: 38828:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e60c9b7db00) refcount = 3 [38655.911335] LustreError: 38828:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [38655.911341] LustreError: 38828:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5fd42bae00/0x8223d61297ae1003 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5998e2784 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [38657.305981] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [38729.424019] LustreError: 16852:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [38729.531269] LustreError: 16852:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 74601 previous similar messages [38738.412313] Lustre: oak-MDT0000: Client 2bea59bc-0913-b755-9506-7536fccd13c0 (at 10.8.4.33@o2ib6) reconnecting [38738.532045] Lustre: Skipped 1 previous similar message [38922.442726] Lustre: oak-MDT0000: Client 121473de-1262-24a0-5ff6-c8e0f00704b7 (at 10.8.18.33@o2ib6) reconnecting [38922.563489] Lustre: Skipped 104 previous similar messages [38968.023355] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [38968.131635] Lustre: Skipped 110 previous similar messages [39001.515213] perf: interrupt took too long (3161 > 3148), lowering kernel.perf_event_max_sample_rate to 63000 [39054.071524] Lustre: oak-MDT0000: Client 4947f37a-215f-b278-6810-6a5656cf3f07 (at 10.8.10.24@o2ib6) reconnecting [39182.107013] Lustre: oak-MDT0000: Client 2eff1e1b-e43a-5bad-f945-ee62707fd637 (at 10.8.2.13@o2ib6) reconnecting [39182.226735] Lustre: Skipped 279 previous similar messages [39276.949059] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [39277.112502] LustreError: Skipped 1 previous similar message [39277.179231] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567211228, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5add75fc00/0x8223d612a5f9d8f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d59a7b67f9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [39277.181278] LustreError: 38941:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5fee0855c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [39277.181280] LustreError: 38941:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5fee0855c0) refcount = 3 [39277.181282] LustreError: 38941:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [39277.181286] LustreError: 38941:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5add75fc00/0x8223d612a5f9d8f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d59a7b67f9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [39278.453893] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [39335.149934] LustreError: 16930:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [39335.149936] LustreError: 16609:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [39335.149938] LustreError: 16609:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 56049 previous similar messages [39335.485522] LustreError: 16930:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 135 previous similar messages [39449.099810] Lustre: oak-MDT0000: Client 0b989bfa-d47e-65b6-f49b-cb9651f8153e (at 10.8.11.15@o2ib6) reconnecting [39449.220572] Lustre: Skipped 304 previous similar messages [39568.559945] Lustre: oak-MDT0000: Connection restored to c31ed026-49ad-bfde-d67f-c9b59a87cc6d (at 10.8.10.22@o2ib6) [39568.683828] Lustre: Skipped 698 previous similar messages [39585.564241] LustreError: 38982:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e60bf350840) refcount = 2 [39585.720370] LustreError: 38982:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [39915.318115] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [39915.481548] LustreError: Skipped 1 previous similar message [39915.548260] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567211866, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e69b7e09200/0x8223d612afdd55a2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d59b411d09 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [39915.548488] LustreError: 39051:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e69dff8b080) refcount nonzero (2) after lock cleanup; forcing cleanup. [39915.548489] LustreError: 39051:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [39915.548492] LustreError: 39051:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e69dff8b080) refcount = 3 [39915.548493] LustreError: 39051:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [39915.548498] LustreError: 39051:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e69b7e09200/0x8223d612afdd55a2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d59b411d09 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [39915.548499] LustreError: 39051:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [39917.068546] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [39954.633089] LustreError: 16852:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [39954.633090] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [39954.633093] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 32631 previous similar messages [39954.968374] LustreError: 16852:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 247 previous similar messages [39966.772147] Lustre: oak-MDT0000: Client 7330e9a1-aa62-4461-d540-34178f96e8cb (at 10.8.11.9@o2ib6) reconnecting [39966.891868] Lustre: Skipped 598 previous similar messages [40193.558273] Lustre: oak-MDT0000: Connection restored to 88809a86-cbd6-8d34-12e6-2bcab8d41b97 (at 10.8.10.29@o2ib6) [40193.682152] Lustre: Skipped 716 previous similar messages [40225.847359] LustreError: 39096:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e54d8f54780) refcount = 2 [40226.003484] LustreError: 39096:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [40534.716864] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [40534.880287] LustreError: Skipped 1 previous similar message [40534.947047] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567212486, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4bb1f92800/0x8223d612b7ec0bb6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d59c2d2e32 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [40534.947450] LustreError: 39178:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e584f855d40) refcount nonzero (2) after lock cleanup; forcing cleanup. [40534.947452] LustreError: 39178:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [40534.947456] LustreError: 39178:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e584f855d40) refcount = 3 [40534.947458] LustreError: 39178:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [40534.947465] LustreError: 39178:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4bb1f92800/0x8223d612b7ec0bb6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d59c2d2e32 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [40534.947467] LustreError: 39178:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [40536.467606] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [40572.144370] Lustre: oak-MDT0000: Client f57c2fff-051c-1ac2-8850-8c55e2caa91a (at 10.8.3.4@o2ib6) reconnecting [40572.263051] Lustre: Skipped 445 previous similar messages [40573.161516] LustreError: 37925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [40573.268846] LustreError: 37925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 39024 previous similar messages [40881.041468] LustreError: 39244:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6093d9d440) refcount = 2 [40881.197594] LustreError: 39244:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [40881.297616] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [40881.405909] Lustre: Skipped 238 previous similar messages [41172.426914] Lustre: oak-MDT0000: Client 4fba8908-ef76-b7eb-c9fc-aacc3bf0191f (at 10.8.2.34@o2ib6) reconnecting [41172.546667] Lustre: Skipped 97 previous similar messages [41188.303147] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [41188.466549] LustreError: Skipped 1 previous similar message [41188.533244] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567213139, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5f39ec3a00/0x8223d612bdabb84c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d59cdb285a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [41188.533646] LustreError: 39396:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e692921dec0) refcount nonzero (2) after lock cleanup; forcing cleanup. [41188.533650] LustreError: 39396:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [41188.533656] LustreError: 39396:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e692921dec0) refcount = 3 [41188.533660] LustreError: 39396:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [41188.533672] LustreError: 39396:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5f39ec3a00/0x8223d612bdabb84c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d59cdb285a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [41188.533675] LustreError: 39396:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [41190.053556] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [41201.275835] LustreError: 34447:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [41201.383091] LustreError: 34447:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 51598 previous similar messages [41486.294123] Lustre: oak-MDT0000: Connection restored to b1b294f1-0ee9-d966-2d69-01a1d0897c58 (at 10.8.13.17@o2ib6) [41486.418008] Lustre: Skipped 430 previous similar messages [41507.020504] LustreError: 39480:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e62eecf7440) refcount = 2 [41507.176629] LustreError: 39480:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [41728.558265] Lustre: oak-MDT0000: haven't heard from client 1a90694a-3084-2825-cdb6-25c2cfb55e50 (at 10.9.110.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e695cfa9000, cur 1567213980 expire 1567213830 last 1567213753 [41777.940344] LNet: Service thread pid 37939 was inactive for 200.47s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [41778.143285] Pid: 37939, comm: mdt_rdpg01_057 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 [41778.270300] Call Trace: [41778.299554] [<ffffffff9b87a39a>] __wait_on_buffer+0x2a/0x30 [41778.367408] [<ffffffffc138017c>] ldiskfs_bread+0x7c/0xc0 [ldiskfs] [41778.442556] [<ffffffffc133c0ca>] __ldiskfs_read_dirblock+0x4a/0x400 [ldiskfs] [41778.529133] [<ffffffffc133c4c0>] htree_dirblock_to_tree+0x40/0x190 [ldiskfs] [41778.614668] [<ffffffffc133e6a1>] ldiskfs_htree_fill_tree+0x201/0x2f0 [ldiskfs] [41778.702278] [<ffffffffc1331b0c>] ldiskfs_readdir+0x61c/0x850 [ldiskfs] [41778.781554] [<ffffffffc144fece>] osd_ldiskfs_it_fill+0xbe/0x260 [osd_ldiskfs] [41778.868130] [<ffffffffc145023e>] osd_it_ea_next+0xce/0x180 [osd_ldiskfs] [41778.949501] [<ffffffffc166f987>] lod_it_next+0x27/0x90 [lod] [41779.018392] [<ffffffffc16e966f>] mdd_dir_page_build+0xaf/0x290 [mdd] [41779.095603] [<ffffffffc0da2e30>] dt_index_walk+0x1a0/0x430 [obdclass] [41779.173865] [<ffffffffc16eb24f>] mdd_readpage+0x25f/0x5a0 [mdd] [41779.245877] [<ffffffffc1584a1a>] mdt_readpage+0x63a/0x880 [mdt] [41779.317885] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [41779.400349] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [41779.492157] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [41779.567319] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [41779.625795] [<ffffffff9bd77c37>] ret_from_fork_nospec_end+0x0/0x39 [41779.700916] [<ffffffffffffffff>] 0xffffffffffffffff [41779.760456] LustreError: dumping log to /tmp/lustre-log.1567214031.37939 [41783.391057] Lustre: oak-MDT0000: Client bb23cb3a-939d-d745-aed1-c5c9a9b7892f (at 10.8.10.11@o2ib6) reconnecting [41783.511818] Lustre: Skipped 607 previous similar messages [41818.754875] LustreError: 37923:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [41818.862132] LustreError: 37923:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 36091 previous similar messages [41842.457372] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [41842.620777] LustreError: Skipped 1 previous similar message [41842.687499] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567213793, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5295763000/0x8223d612c52d6df9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d59dc6bf2d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [41842.687666] LustreError: 39622:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e572faa3a40) refcount nonzero (2) after lock cleanup; forcing cleanup. [41842.687668] LustreError: 39622:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [41842.687671] LustreError: 39622:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e572faa3a40) refcount = 3 [41842.687672] LustreError: 39622:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [41842.687677] LustreError: 39622:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5295763000/0x8223d612c52d6df9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d59dc6bf2d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [41842.687678] LustreError: 39622:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [41844.209314] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [41941.785152] LNet: Service thread pid 37939 completed after 364.31s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [42100.233394] Lustre: oak-MDT0000: Connection restored to 7aad685c-f631-4b7e-cf4e-452b69ed5d26 (at 10.8.2.18@o2ib6) [42100.356229] Lustre: Skipped 524 previous similar messages [42152.085395] LustreError: 39682:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56e6037d40) refcount = 1 [42392.217926] Lustre: oak-MDT0000: Client 92b1f865-efd9-b34c-bfa7-a050ff38a714 (at 10.8.10.32@o2ib6) reconnecting [42392.338688] Lustre: Skipped 513 previous similar messages [42447.213204] LustreError: 33399:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [42447.320448] LustreError: 33399:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 41949 previous similar messages [42470.022136] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [42470.185541] LustreError: Skipped 1 previous similar message [42470.252232] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567214421, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6926a1ca00/0x8223d612cb5ab51a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d59ea607b3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [42470.252547] LustreError: 39751:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e60d9246900) refcount nonzero (2) after lock cleanup; forcing cleanup. [42470.252549] LustreError: 39751:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [42470.252551] LustreError: 39751:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e60d9246900) refcount = 3 [42470.252553] LustreError: 39751:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [42470.252558] LustreError: 39751:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6926a1ca00/0x8223d612cb5ab51a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d59ea607b3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [42471.651771] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [42709.275611] Lustre: oak-MDT0000: Connection restored to 659c113f-42e9-1358-24fe-aeba5450befb (at 10.8.10.4@o2ib6) [42709.398468] Lustre: Skipped 565 previous similar messages [42780.047267] LustreError: 39792:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e631acbdd40) refcount = 1 [42999.307311] Lustre: oak-MDT0000: Client 9a9ff06e-b621-ceac-e7d1-f8dc3b68236d (at 10.8.2.14@o2ib6) reconnecting [42999.427035] Lustre: Skipped 576 previous similar messages [43051.994870] LustreError: 16927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [43052.102121] LustreError: 16927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 45340 previous similar messages [43094.935909] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [43095.099314] LustreError: Skipped 1 previous similar message [43095.166003] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567215046, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f52458000/0x8223d612d31e60fe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d59f85197b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [43095.166215] LustreError: 39832:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e58487ac480) refcount nonzero (2) after lock cleanup; forcing cleanup. [43095.166216] LustreError: 39832:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [43095.166219] LustreError: 39832:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e58487ac480) refcount = 3 [43095.166220] LustreError: 39832:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [43095.166225] LustreError: 39832:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f52458000/0x8223d612d31e60fe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d59f85197b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [43096.565554] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [43309.474370] Lustre: oak-MDT0000: Connection restored to 7aad685c-f631-4b7e-cf4e-452b69ed5d26 (at 10.8.2.18@o2ib6) [43309.597209] Lustre: Skipped 961 previous similar messages [43407.453011] LustreError: 39885:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e4f1c7e1d40) refcount = 2 [43602.744185] Lustre: oak-MDT0000: Client 0f7149bc-dcde-9a20-a715-bf762be6aba6 (at 10.8.13.21@o2ib6) reconnecting [43602.864950] Lustre: Skipped 1274 previous similar messages [43652.036653] LustreError: 16859:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [43652.143921] LustreError: 16859:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 39485 previous similar messages [43713.581621] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [43713.745019] LustreError: Skipped 1 previous similar message [43713.811703] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567215665, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6568e99c00/0x8223d612d9a6b8d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a06cdfa4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [43713.811949] LustreError: 39920:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64371d6000) refcount nonzero (2) after lock cleanup; forcing cleanup. [43713.811950] LustreError: 39920:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [43713.811953] LustreError: 39920:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64371d6000) refcount = 3 [43713.811955] LustreError: 39920:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [43713.811960] LustreError: 39920:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6568e99c00/0x8223d612d9a6b8d4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a06cdfa4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [43715.211259] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [43911.005873] Lustre: oak-MDT0000: Connection restored to 2bea59bc-0913-b755-9506-7536fccd13c0 (at 10.8.4.33@o2ib6) [43911.128725] Lustre: Skipped 958 previous similar messages [44022.594759] LustreError: 40035:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64f1157ec0) refcount = 2 [44022.750890] LustreError: 40035:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [44189.498082] LNet: Service thread pid 16876 was inactive for 200.27s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [44189.701001] Pid: 16876, comm: mdt_rdpg01_022 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 [44189.827996] Call Trace: [44189.857240] [<ffffffff9b87a39a>] __wait_on_buffer+0x2a/0x30 [44189.925067] [<ffffffffc138017c>] ldiskfs_bread+0x7c/0xc0 [ldiskfs] [44190.000182] [<ffffffffc133c0ca>] __ldiskfs_read_dirblock+0x4a/0x400 [ldiskfs] [44190.086732] [<ffffffffc133c4c0>] htree_dirblock_to_tree+0x40/0x190 [ldiskfs] [44190.172235] [<ffffffffc133e6a1>] ldiskfs_htree_fill_tree+0x201/0x2f0 [ldiskfs] [44190.259826] [<ffffffffc1331b0c>] ldiskfs_readdir+0x61c/0x850 [ldiskfs] [44190.339115] [<ffffffffc144fece>] osd_ldiskfs_it_fill+0xbe/0x260 [osd_ldiskfs] [44190.425695] [<ffffffffc145023e>] osd_it_ea_next+0xce/0x180 [osd_ldiskfs] [44190.507055] [<ffffffffc166f987>] lod_it_next+0x27/0x90 [lod] [44190.575948] [<ffffffffc16e966f>] mdd_dir_page_build+0xaf/0x290 [mdd] [44190.653188] [<ffffffffc0da2e30>] dt_index_walk+0x1a0/0x430 [obdclass] [44190.731449] [<ffffffffc16eb24f>] mdd_readpage+0x25f/0x5a0 [mdd] [44190.803448] [<ffffffffc1584a1a>] mdt_readpage+0x63a/0x880 [mdt] [44190.875461] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [44190.957911] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [44191.049686] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [44191.124831] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [44191.183301] [<ffffffff9bd77c37>] ret_from_fork_nospec_end+0x0/0x39 [44191.258411] [<ffffffffffffffff>] 0xffffffffffffffff [44191.317934] LustreError: dumping log to /tmp/lustre-log.1567216442.16876 [44203.791796] Lustre: oak-MDT0000: Client 45b0fd57-f055-545d-5e08-471622d7d63a (at 10.8.2.5@o2ib6) reconnecting [44203.910473] Lustre: Skipped 805 previous similar messages [44252.078741] LustreError: 34448:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [44252.186019] LustreError: 34448:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 38758 previous similar messages [44328.240265] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [44328.403683] LustreError: Skipped 1 previous similar message [44328.470380] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567216279, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e695a2a0400/0x8223d612e047c99f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a1700869 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [44328.470555] LustreError: 40114:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6076150a80) refcount nonzero (2) after lock cleanup; forcing cleanup. [44328.470557] LustreError: 40114:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [44328.470559] LustreError: 40114:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6076150a80) refcount = 3 [44328.470561] LustreError: 40114:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [44328.470567] LustreError: 40114:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e695a2a0400/0x8223d612e047c99f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a1700869 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [44328.470568] LustreError: 40114:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [44329.990687] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [44412.585460] LNet: Service thread pid 16876 completed after 423.35s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [44511.551319] Lustre: oak-MDT0000: Connection restored to 315dcdcc-f238-a1dc-aff3-ff51fc65dabe (at 10.8.13.13@o2ib6) [44511.675202] Lustre: Skipped 748 previous similar messages [44635.903332] LustreError: 40183:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e604fff0180) refcount = 2 [44636.059450] LustreError: 40183:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [44733.600151] Lustre: oak-MDT0000: haven't heard from client 52f9db0a-72a2-d383-4256-6f5d15b0a631 (at 10.9.113.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966571c00, cur 1567216985 expire 1567216835 last 1567216758 [44803.933044] Lustre: oak-MDT0000: Client 00c02d24-a599-f241-a134-7950e3df9301 (at 10.8.12.24@o2ib6) reconnecting [44804.053814] Lustre: Skipped 726 previous similar messages [44852.104307] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [44852.211550] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 47262 previous similar messages [44945.415982] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [44945.579388] LustreError: Skipped 1 previous similar message [44945.646086] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567216896, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6653f54800/0x8223d612e8bacf80 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a27a6c6f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [44945.646358] LustreError: 40216:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6921a0fec0) refcount nonzero (2) after lock cleanup; forcing cleanup. [44945.646360] LustreError: 40216:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [44945.646362] LustreError: 40216:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6921a0fec0) refcount = 3 [44945.646364] LustreError: 40216:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [44945.646368] LustreError: 40216:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6653f54800/0x8223d612e8bacf80 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a27a6c6f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [44945.646370] LustreError: 40216:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [44947.166343] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [45123.177150] Lustre: oak-MDT0000: Connection restored to d6708f73-6080-dbfe-81af-32f3b24e006b (at 10.8.12.30@o2ib6) [45123.301034] Lustre: Skipped 760 previous similar messages [45405.349421] Lustre: oak-MDT0000: Client 4ecd0193-89bf-ec09-52b1-fa3f000de1f6 (at 10.8.12.15@o2ib6) reconnecting [45405.470274] Lustre: Skipped 675 previous similar messages [45452.317573] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [45452.424815] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 52593 previous similar messages [45572.728813] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [45572.892218] LustreError: Skipped 1 previous similar message [45572.958916] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567217524, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e69445f7800/0x8223d612f1fe295e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a3685e92 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [45572.959122] LustreError: 40301:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5aa352d980) refcount nonzero (2) after lock cleanup; forcing cleanup. [45572.959125] LustreError: 40301:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5aa352d980) refcount = 3 [45572.959127] LustreError: 40301:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [45572.959132] LustreError: 40301:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e69445f7800/0x8223d612f1fe295e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a3685e92 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [45574.233549] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [45656.691503] perf: interrupt took too long (3955 > 3951), lowering kernel.perf_event_max_sample_rate to 50000 [45727.641236] Lustre: oak-MDT0000: Connection restored to 47fd202d-4cdb-b3f0-258d-80213c171f8f (at 10.8.6.6@o2ib6) [45727.763079] Lustre: Skipped 554 previous similar messages [45895.950010] LustreError: 40348:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e6775afab40) refcount = 2 [46013.136054] Lustre: oak-MDT0000: Client 3647e301-3bcd-4b7a-f688-ac3a2cfc9e91 (at 10.8.11.31@o2ib6) reconnecting [46013.256813] Lustre: Skipped 589 previous similar messages [46052.431567] LustreError: 16613:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [46052.538809] LustreError: 16613:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 43103 previous similar messages [46203.608673] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [46203.772075] LustreError: Skipped 1 previous similar message [46203.838762] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567218154, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c558d0e00/0x8223d612fca13d77 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a4573722 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [46203.838978] LustreError: 40383:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64caa7d380) refcount nonzero (2) after lock cleanup; forcing cleanup. [46203.838980] LustreError: 40383:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [46203.838982] LustreError: 40383:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64caa7d380) refcount = 3 [46203.838984] LustreError: 40383:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [46203.838989] LustreError: 40383:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c558d0e00/0x8223d612fca13d77 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a4573722 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [46205.238285] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [46332.894109] Lustre: oak-MDT0000: Connection restored to 55cc63cc-00e0-9d99-735c-4552540f5bf5 (at 10.8.10.9@o2ib6) [46333.016949] Lustre: Skipped 750 previous similar messages [46638.779477] Lustre: oak-MDT0000: Client 689b0a06-03db-e56d-5cd7-2b06de17cd12 (at 10.8.11.7@o2ib6) reconnecting [46638.899218] Lustre: Skipped 518 previous similar messages [46653.533537] LustreError: 16605:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [46653.640780] LustreError: 16605:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 27198 previous similar messages [46883.787215] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [46883.950614] LustreError: Skipped 1 previous similar message [46884.017302] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567218835, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5619d8cc00/0x8223d61304ee53cd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a53ed711 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [46884.017543] LustreError: 40462:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5e6cc77500) refcount nonzero (2) after lock cleanup; forcing cleanup. [46884.017546] LustreError: 40462:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5e6cc77500) refcount = 3 [46884.017548] LustreError: 40462:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [46884.017552] LustreError: 40462:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5619d8cc00/0x8223d61304ee53cd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a53ed711 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [46885.291920] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [46940.823706] Lustre: oak-MDT0000: Connection restored to ecb6f4c2-0cc7-9e7a-0f4c-c7f3f1044ecb (at 10.8.10.7@o2ib6) [46940.946545] Lustre: Skipped 172 previous similar messages [47193.115338] LustreError: 40512:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5d2c60acc0) refcount = 2 [47277.211858] Lustre: oak-MDT0000: Client 5111f45b-3105-5779-d07d-234e50dd05e6 (at 10.8.12.28@o2ib6) reconnecting [47277.332622] Lustre: Skipped 75 previous similar messages [47291.176118] LustreError: 33468:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [47291.283365] LustreError: 33468:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 33464 previous similar messages [47372.139875] LNet: Service thread pid 37934 was inactive for 242.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [47372.342799] Pid: 37934, comm: mdt_rdpg01_054 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 [47372.469795] Call Trace: [47372.499049] [<ffffffff9b87a39a>] __wait_on_buffer+0x2a/0x30 [47372.566893] [<ffffffffc138017c>] ldiskfs_bread+0x7c/0xc0 [ldiskfs] [47372.642024] [<ffffffffc133c0ca>] __ldiskfs_read_dirblock+0x4a/0x400 [ldiskfs] [47372.728578] [<ffffffffc133c4c0>] htree_dirblock_to_tree+0x40/0x190 [ldiskfs] [47372.814095] [<ffffffffc133e6a1>] ldiskfs_htree_fill_tree+0x201/0x2f0 [ldiskfs] [47372.901691] [<ffffffffc1331b0c>] ldiskfs_readdir+0x61c/0x850 [ldiskfs] [47372.980968] [<ffffffffc144fece>] osd_ldiskfs_it_fill+0xbe/0x260 [osd_ldiskfs] [47373.067545] [<ffffffffc145023e>] osd_it_ea_next+0xce/0x180 [osd_ldiskfs] [47373.148935] [<ffffffffc166f987>] lod_it_next+0x27/0x90 [lod] [47373.217839] [<ffffffffc16e966f>] mdd_dir_page_build+0xaf/0x290 [mdd] [47373.295073] [<ffffffffc0da2e30>] dt_index_walk+0x1a0/0x430 [obdclass] [47373.373367] [<ffffffffc16eb24f>] mdd_readpage+0x25f/0x5a0 [mdd] [47373.445427] [<ffffffffc1584a1a>] mdt_readpage+0x63a/0x880 [mdt] [47373.517477] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [47373.599950] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [47373.691776] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [47373.766952] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [47373.825443] [<ffffffff9bd77c37>] ret_from_fork_nospec_end+0x0/0x39 [47373.900575] [<ffffffffffffffff>] 0xffffffffffffffff [47373.960132] LustreError: dumping log to /tmp/lustre-log.1567219625.37934 [47478.720375] LNet: Service thread pid 37934 completed after 349.18s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [47502.270080] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [47502.433498] LustreError: Skipped 1 previous similar message [47502.500207] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567219453, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e64c3f79000/0x8223d61309f0bc07 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a62dd0d3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [47502.500378] LustreError: 40542:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64968449c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [47502.500380] LustreError: 40542:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [47502.500385] LustreError: 40542:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64968449c0) refcount = 3 [47502.500388] LustreError: 40542:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [47502.500394] LustreError: 40542:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e64c3f79000/0x8223d61309f0bc07 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a62dd0d3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [47503.899802] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [47542.301751] Lustre: oak-MDT0000: Connection restored to 56556a24-1a3f-ba58-bbd5-aad33c854cc4 (at 10.8.4.20@o2ib6) [47542.424596] Lustre: Skipped 128 previous similar messages [47566.645530] Lustre: oak-MDT0000: haven't heard from client b67965cf-4a75-2de2-5973-72f9b63dcc08 (at 171.65.177.202@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6967361000, cur 1567219818 expire 1567219668 last 1567219591 [47811.639100] LustreError: 40607:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5db6f5d5c0) refcount = 2 [47878.072046] Lustre: oak-MDT0000: Client b04a309f-45a4-e137-125c-79ca0c8fd4e4 (at 10.8.12.4@o2ib6) reconnecting [47878.191766] Lustre: Skipped 321 previous similar messages [47907.124763] LustreError: 16928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [47907.232005] LustreError: 16928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 22101 previous similar messages [48133.202928] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [48133.366333] LustreError: Skipped 1 previous similar message [48133.433031] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567220084, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e698f2e1a00/0x8223d6130f9900c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a70afd02 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [48133.433229] LustreError: 40637:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5b1270fe00) refcount nonzero (2) after lock cleanup; forcing cleanup. [48133.433231] LustreError: 40637:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [48133.433233] LustreError: 40637:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5b1270fe00) refcount = 3 [48133.433235] LustreError: 40637:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [48133.433242] LustreError: 40637:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e698f2e1a00/0x8223d6130f9900c0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a70afd02 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [48134.832555] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [48147.849757] Lustre: oak-MDT0000: Connection restored to a67313f2-9233-a027-41ef-9de0290e5d3e (at 10.8.13.4@o2ib6) [48147.972606] Lustre: Skipped 365 previous similar messages [48497.708133] Lustre: oak-MDT0000: Client de7f395d-d6fe-fc19-7c12-cbadef5a17bc (at 10.8.4.35@o2ib6) reconnecting [48497.827855] Lustre: Skipped 231 previous similar messages [48519.026147] LustreError: 40715:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e63935c3ec0) refcount = 2 [48519.182278] LustreError: 40715:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [48543.640922] LustreError: 34442:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [48543.748224] LustreError: 34442:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 27084 previous similar messages [48777.736498] Lustre: oak-MDT0000: Connection restored to 36ee5e48-1b22-27dc-beca-b392c0250a2d (at 10.8.4.26@o2ib6) [48777.859340] Lustre: Skipped 212 previous similar messages [48853.713009] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [48853.876410] LustreError: Skipped 1 previous similar message [48853.943100] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567220805, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51573e9200/0x8223d61315cacb66 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a8298bb9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [48853.943378] LustreError: 40765:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5eadc1e540) refcount nonzero (2) after lock cleanup; forcing cleanup. [48853.943379] LustreError: 40765:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [48853.943382] LustreError: 40765:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5eadc1e540) refcount = 3 [48853.943384] LustreError: 40765:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [48853.943389] LustreError: 40765:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51573e9200/0x8223d61315cacb66 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a8298bb9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [48853.943390] LustreError: 40765:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [48855.463393] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [49081.678606] INFO: task mdt00_000:16178 blocked for more than 120 seconds. [49081.759844] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [49081.853558] mdt00_000 D ffff8e69e5ca0000 0 16178 2 0x00000080 [49081.938134] Call Trace: [49081.967373] [<ffffffff9bd6be39>] schedule_preempt_disabled+0x29/0x70 [49082.044449] [<ffffffff9bd69db7>] __mutex_lock_slowpath+0xc7/0x1d0 [49082.118406] [<ffffffff9bd6919f>] mutex_lock+0x1f/0x2f [49082.179893] [<ffffffffc0485508>] __jbd2_log_wait_for_space+0xc8/0x1f0 [jbd2] [49082.265290] [<ffffffffc047f3d3>] add_transaction_credits+0x2d3/0x2f0 [jbd2] [49082.349659] [<ffffffffc13ffd2a>] ? qsd_refresh_usage+0x6a/0x2b0 [lquota] [49082.430894] [<ffffffffc047f5e1>] start_this_handle+0x1a1/0x430 [jbd2] [49082.509011] [<ffffffff9b8b0017>] ? dqget+0x1a7/0x450 [49082.569447] [<ffffffff9b81d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0 [49082.641321] [<ffffffffc047fa93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] [49082.720490] [<ffffffffc138538f>] ? ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49082.805887] [<ffffffffc1366189>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [49082.894406] [<ffffffffc138538f>] ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49082.977721] [<ffffffff9b85ff34>] evict+0xb4/0x180 [49083.035032] [<ffffffff9b86083c>] iput+0xfc/0x190 [49083.091314] [<ffffffffc1452d73>] osd_object_delete+0x1d3/0x330 [osd_ldiskfs] [49083.176731] [<ffffffffc0d9d548>] lu_object_free.isra.31+0xa8/0x1c0 [obdclass] [49083.263174] [<ffffffffc0da0c15>] lu_object_put+0xc5/0x3e0 [obdclass] [49083.340266] [<ffffffffc15a45a9>] mdt_reint_unlink+0x939/0x11d0 [mdt] [49083.417344] [<ffffffffc15a7ca3>] mdt_reint_rec+0x83/0x210 [mdt] [49083.489221] [<ffffffffc158918b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] [49083.567340] [<ffffffffc1594fa7>] mdt_reint+0x67/0x140 [mdt] [49083.635111] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [49083.717405] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [49083.809060] [<ffffffffc0fd0688>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [49083.890315] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [49083.965333] [<ffffffffc0fd6d10>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [49084.053852] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [49084.112207] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49084.185122] [<ffffffff9bd77c37>] ret_from_fork_nospec_begin+0x21/0x21 [49084.263232] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49084.336145] INFO: task mdt00_001:16179 blocked for more than 120 seconds. [49084.417376] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [49084.511088] mdt00_001 D ffff8e69e5ca4100 0 16179 2 0x00000080 [49084.595662] Call Trace: [49084.624905] [<ffffffff9bd6be39>] schedule_preempt_disabled+0x29/0x70 [49084.701981] [<ffffffff9bd69db7>] __mutex_lock_slowpath+0xc7/0x1d0 [49084.775932] [<ffffffff9bd6919f>] mutex_lock+0x1f/0x2f [49084.837409] [<ffffffffc0485508>] __jbd2_log_wait_for_space+0xc8/0x1f0 [jbd2] [49084.922807] [<ffffffffc047f3d3>] add_transaction_credits+0x2d3/0x2f0 [jbd2] [49085.007178] [<ffffffffc13ffd2a>] ? qsd_refresh_usage+0x6a/0x2b0 [lquota] [49085.088417] [<ffffffffc047f5e1>] start_this_handle+0x1a1/0x430 [jbd2] [49085.166534] [<ffffffff9b8b0017>] ? dqget+0x1a7/0x450 [49085.226966] [<ffffffff9b81d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0 [49085.298855] [<ffffffffc047fa93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] [49085.378015] [<ffffffffc138538f>] ? ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49085.463428] [<ffffffffc1366189>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [49085.551959] [<ffffffffc138538f>] ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49085.635275] [<ffffffff9b85ff34>] evict+0xb4/0x180 [49085.692587] [<ffffffff9b86083c>] iput+0xfc/0x190 [49085.748874] [<ffffffffc1452d73>] osd_object_delete+0x1d3/0x330 [osd_ldiskfs] [49085.834282] [<ffffffffc0d9d548>] lu_object_free.isra.31+0xa8/0x1c0 [obdclass] [49085.920727] [<ffffffffc0da0c15>] lu_object_put+0xc5/0x3e0 [obdclass] [49085.997808] [<ffffffffc15a45a9>] mdt_reint_unlink+0x939/0x11d0 [mdt] [49086.074884] [<ffffffffc15a7ca3>] mdt_reint_rec+0x83/0x210 [mdt] [49086.146763] [<ffffffffc158918b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] [49086.224882] [<ffffffffc1594fa7>] mdt_reint+0x67/0x140 [mdt] [49086.292616] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [49086.374907] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [49086.466585] [<ffffffffc0fd0688>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [49086.547822] [<ffffffff9b6d7c40>] ? wake_up_state+0x20/0x20 [49086.614515] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [49086.689528] [<ffffffffc0fd6d10>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [49086.778042] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [49086.836393] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49086.909307] [<ffffffff9bd77c37>] ret_from_fork_nospec_begin+0x21/0x21 [49086.987418] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49087.060330] INFO: task mdt00_002:16180 blocked for more than 120 seconds. [49087.141561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [49087.235274] mdt00_002 D ffff8e69e5ca5140 0 16180 2 0x00000080 [49087.319854] Call Trace: [49087.349087] [<ffffffff9bd6be39>] schedule_preempt_disabled+0x29/0x70 [49087.426161] [<ffffffff9bd69db7>] __mutex_lock_slowpath+0xc7/0x1d0 [49087.500110] [<ffffffff9bd6919f>] mutex_lock+0x1f/0x2f [49087.561601] [<ffffffffc0485508>] __jbd2_log_wait_for_space+0xc8/0x1f0 [jbd2] [49087.646998] [<ffffffffc047f3d3>] add_transaction_credits+0x2d3/0x2f0 [jbd2] [49087.731366] [<ffffffffc13ffd2a>] ? qsd_refresh_usage+0x6a/0x2b0 [lquota] [49087.812604] [<ffffffffc047f5e1>] start_this_handle+0x1a1/0x430 [jbd2] [49087.890721] [<ffffffff9b8b0017>] ? dqget+0x1a7/0x450 [49087.951151] [<ffffffff9b81d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0 [49088.023026] [<ffffffffc047fa93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] [49088.102182] [<ffffffffc138538f>] ? ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49088.187580] [<ffffffffc1366189>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [49088.276098] [<ffffffffc138538f>] ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49088.359415] [<ffffffff9b85ff34>] evict+0xb4/0x180 [49088.416725] [<ffffffff9b86083c>] iput+0xfc/0x190 [49088.473007] [<ffffffffc1452d73>] osd_object_delete+0x1d3/0x330 [osd_ldiskfs] [49088.558427] [<ffffffffc0d9d548>] lu_object_free.isra.31+0xa8/0x1c0 [obdclass] [49088.644875] [<ffffffffc0da0c15>] lu_object_put+0xc5/0x3e0 [obdclass] [49088.721955] [<ffffffffc15a45a9>] mdt_reint_unlink+0x939/0x11d0 [mdt] [49088.799044] [<ffffffffc15a7ca3>] mdt_reint_rec+0x83/0x210 [mdt] [49088.870929] [<ffffffffc158918b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] [49088.949046] [<ffffffffc1594fa7>] mdt_reint+0x67/0x140 [mdt] [49089.016786] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [49089.099078] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [49089.190733] [<ffffffffc0fd0688>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [49089.271977] [<ffffffff9b6d7c52>] ? default_wake_function+0x12/0x20 [49089.346982] [<ffffffff9b6ccceb>] ? __wake_up_common+0x5b/0x90 [49089.416793] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [49089.491805] [<ffffffffc0fd6d10>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [49089.580321] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [49089.638672] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49089.711582] [<ffffffff9bd77c37>] ret_from_fork_nospec_begin+0x21/0x21 [49089.789705] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49089.862623] INFO: task mdt01_001:16182 blocked for more than 120 seconds. [49089.943855] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [49090.037567] mdt01_001 D ffff8e58d44b0000 0 16182 2 0x00000080 [49090.122139] Call Trace: [49090.151372] [<ffffffff9bd6be39>] schedule_preempt_disabled+0x29/0x70 [49090.228445] [<ffffffff9bd69db7>] __mutex_lock_slowpath+0xc7/0x1d0 [49090.302398] [<ffffffff9bd6919f>] mutex_lock+0x1f/0x2f [49090.363889] [<ffffffffc0485508>] __jbd2_log_wait_for_space+0xc8/0x1f0 [jbd2] [49090.449282] [<ffffffffc047f3d3>] add_transaction_credits+0x2d3/0x2f0 [jbd2] [49090.533651] [<ffffffffc13ffd2a>] ? qsd_refresh_usage+0x6a/0x2b0 [lquota] [49090.614900] [<ffffffffc047f5e1>] start_this_handle+0x1a1/0x430 [jbd2] [49090.693026] [<ffffffff9b8b0017>] ? dqget+0x1a7/0x450 [49090.753456] [<ffffffff9b81d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0 [49090.825331] [<ffffffffc047fa93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] [49090.904498] [<ffffffffc138538f>] ? ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49090.989898] [<ffffffffc1366189>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [49091.078415] [<ffffffffc138538f>] ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49091.161734] [<ffffffff9b85ff34>] evict+0xb4/0x180 [49091.219049] [<ffffffff9b86083c>] iput+0xfc/0x190 [49091.275324] [<ffffffffc1452d73>] osd_object_delete+0x1d3/0x330 [osd_ldiskfs] [49091.360738] [<ffffffffc0d9d548>] lu_object_free.isra.31+0xa8/0x1c0 [obdclass] [49091.447184] [<ffffffffc0da0c15>] lu_object_put+0xc5/0x3e0 [obdclass] [49091.524266] [<ffffffffc15a45a9>] mdt_reint_unlink+0x939/0x11d0 [mdt] [49091.601342] [<ffffffffc15a7ca3>] mdt_reint_rec+0x83/0x210 [mdt] [49091.673218] [<ffffffffc158918b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] [49091.751332] [<ffffffffc1594fa7>] mdt_reint+0x67/0x140 [mdt] [49091.819072] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [49091.901366] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [49091.993030] [<ffffffffc0fd0688>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [49092.074287] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [49092.149303] [<ffffffffc0fd6d10>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [49092.237828] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [49092.296181] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49092.369104] [<ffffffff9bd77c37>] ret_from_fork_nospec_begin+0x21/0x21 [49092.447230] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49092.520155] INFO: task mdt01_002:16183 blocked for more than 120 seconds. [49092.601400] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [49092.695124] mdt01_002 D ffff8e58d44b1040 0 16183 2 0x00000080 [49092.779738] Call Trace: [49092.808984] [<ffffffff9bd6be39>] schedule_preempt_disabled+0x29/0x70 [49092.886062] [<ffffffff9bd69db7>] __mutex_lock_slowpath+0xc7/0x1d0 [49092.960021] [<ffffffff9bd6919f>] mutex_lock+0x1f/0x2f [49093.021510] [<ffffffffc0485508>] __jbd2_log_wait_for_space+0xc8/0x1f0 [jbd2] [49093.106908] [<ffffffffc047f3d3>] add_transaction_credits+0x2d3/0x2f0 [jbd2] [49093.191267] [<ffffffffc13ffd2a>] ? qsd_refresh_usage+0x6a/0x2b0 [lquota] [49093.272506] [<ffffffffc047f5e1>] start_this_handle+0x1a1/0x430 [jbd2] [49093.350624] [<ffffffff9b8b0017>] ? dqget+0x1a7/0x450 [49093.411055] [<ffffffff9b81d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0 [49093.482930] [<ffffffffc047fa93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] [49093.562092] [<ffffffffc138538f>] ? ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49093.647489] [<ffffffffc1366189>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [49093.736008] [<ffffffffc138538f>] ldiskfs_evict_inode+0x16f/0x630 [ldiskfs] [49093.819327] [<ffffffff9b85ff34>] evict+0xb4/0x180 [49093.876641] [<ffffffff9b86083c>] iput+0xfc/0x190 [49093.932915] [<ffffffffc1452d73>] osd_object_delete+0x1d3/0x330 [osd_ldiskfs] [49094.018324] [<ffffffffc0d9d548>] lu_object_free.isra.31+0xa8/0x1c0 [obdclass] [49094.104781] [<ffffffffc0da0c15>] lu_object_put+0xc5/0x3e0 [obdclass] [49094.181863] [<ffffffffc15a45a9>] mdt_reint_unlink+0x939/0x11d0 [mdt] [49094.258941] [<ffffffffc15a7ca3>] mdt_reint_rec+0x83/0x210 [mdt] [49094.330818] [<ffffffffc158918b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] [49094.408938] [<ffffffffc1594fa7>] mdt_reint+0x67/0x140 [mdt] [49094.476687] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [49094.558981] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [49094.650648] [<ffffffffc0fd0688>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [49094.731897] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [49094.806911] [<ffffffffc0fd6d10>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [49094.895423] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [49094.953789] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49095.026702] [<ffffffff9bd77c37>] ret_from_fork_nospec_begin+0x21/0x21 [49095.104824] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49095.177740] INFO: task mdt_rdpg00_000:16184 blocked for more than 120 seconds. [49095.264173] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [49095.357885] mdt_rdpg00_000 D ffff8e58d44b2080 0 16184 2 0x00000080 [49095.442458] Call Trace: [49095.471691] [<ffffffff9bd6be39>] schedule_preempt_disabled+0x29/0x70 [49095.548763] [<ffffffff9bd69db7>] __mutex_lock_slowpath+0xc7/0x1d0 [49095.622716] [<ffffffff9bd6919f>] mutex_lock+0x1f/0x2f [49095.684193] [<ffffffffc0485508>] __jbd2_log_wait_for_space+0xc8/0x1f0 [jbd2] [49095.769588] [<ffffffffc047f3d3>] add_transaction_credits+0x2d3/0x2f0 [jbd2] [49095.853947] [<ffffffffc13710aa>] ? ldiskfs_xattr_trusted_get+0x2a/0x30 [ldiskfs] [49095.943508] [<ffffffff9b86a522>] ? generic_getxattr+0x52/0x70 [49096.013305] [<ffffffffc047f5e1>] start_this_handle+0x1a1/0x430 [jbd2] [49096.091416] [<ffffffff9b81d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0 [49096.163289] [<ffffffffc047fa93>] jbd2__journal_start+0xf3/0x1f0 [jbd2] [49096.242445] [<ffffffffc1459cbe>] ? osd_trans_start+0x1ae/0x460 [osd_ldiskfs] [49096.327845] [<ffffffffc1366189>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [49096.416361] [<ffffffffc1459cbe>] osd_trans_start+0x1ae/0x460 [osd_ldiskfs] [49096.499678] [<ffffffffc141930e>] qmt_trans_start_with_slv+0x25e/0x540 [lquota] [49096.587153] [<ffffffffc1412336>] qmt_dqacq0+0x1a6/0xf00 [lquota] [49096.660090] [<ffffffffc0fc845f>] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [49096.747566] [<ffffffffc1415cfa>] qmt_intent_policy+0x86a/0xe80 [lquota] [49096.827781] [<ffffffffc0fc5b57>] ? lustre_msg_buf+0x17/0x60 [ptlrpc] [49096.904873] [<ffffffffc158c70d>] mdt_intent_opc+0x43d/0xaf0 [mdt] [49096.978846] [<ffffffffc0fc9f90>] ? lustre_swab_ldlm_policy_data+0x30/0x30 [ptlrpc] [49097.070483] [<ffffffffc1594d58>] mdt_intent_policy+0x138/0x320 [mdt] [49097.147572] [<ffffffffc0f782dd>] ldlm_lock_enqueue+0x38d/0x980 [ptlrpc] [49097.227787] [<ffffffffc0fa1c03>] ldlm_handle_enqueue0+0xa83/0x1670 [ptlrpc] [49097.312159] [<ffffffffc0fca010>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [49097.401736] [<ffffffffc1027892>] tgt_enqueue+0x62/0x210 [ptlrpc] [49097.474678] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [49097.557017] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [49097.648727] [<ffffffffc0fd0915>] ? ptlrpc_wait_event+0x325/0x340 [ptlrpc] [49097.731031] [<ffffffff9b6d7c40>] ? wake_up_state+0x20/0x20 [49097.797777] [<ffffffffc0fd750e>] ? ptlrpc_main+0x7fe/0x1e40 [ptlrpc] [49097.874919] [<ffffffffc0fd6d10>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [49097.963462] [<ffffffff9b6c2e81>] ? kthread+0xd1/0xe0 [49098.023920] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49098.096868] [<ffffffff9bd77c37>] ? ret_from_fork_nospec_begin+0x21/0x21 [49098.177089] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [49100.433319] Lustre: oak-MDT0000: Client e10f2417-21db-90f3-e0d4-c0a1b12575cc (at 10.8.10.36@o2ib6) reconnecting [49100.554111] Lustre: Skipped 271 previous similar messages [49189.547806] LustreError: 37931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [49189.655061] LustreError: 37931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 13541 previous similar messages [49379.824587] Lustre: oak-MDT0000: Connection restored to (at 10.8.10.34@o2ib6) [49379.911021] Lustre: Skipped 259 previous similar messages [49517.453268] LNet: Service thread pid 16608 was inactive for 286.98s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [49517.656201] Pid: 16608, comm: mdt_rdpg00_008 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 [49517.783248] Call Trace: [49517.812490] [<ffffffff9b945d03>] get_request+0x243/0x7d0 [49517.877206] [<ffffffff9b948bce>] blk_queue_bio+0xfe/0x400 [49517.942961] [<ffffffff9b946e17>] generic_make_request+0x147/0x380 [49518.017025] [<ffffffff9b9470c0>] submit_bio+0x70/0x150 [49518.079645] [<ffffffff9b87b627>] _submit_bh+0x127/0x160 [49518.143308] [<ffffffff9b87c079>] ll_rw_block+0xa9/0xb0 [49518.205937] [<ffffffffc1380163>] ldiskfs_bread+0x63/0xc0 [ldiskfs] [49518.281085] [<ffffffffc133c0ca>] __ldiskfs_read_dirblock+0x4a/0x400 [ldiskfs] [49518.367681] [<ffffffffc133c4c0>] htree_dirblock_to_tree+0x40/0x190 [ldiskfs] [49518.453191] [<ffffffffc133e6a1>] ldiskfs_htree_fill_tree+0x201/0x2f0 [ldiskfs] [49518.540782] [<ffffffffc1331b0c>] ldiskfs_readdir+0x61c/0x850 [ldiskfs] [49518.620063] [<ffffffffc144fece>] osd_ldiskfs_it_fill+0xbe/0x260 [osd_ldiskfs] [49518.707100] [<ffffffffc145023e>] osd_it_ea_next+0xce/0x180 [osd_ldiskfs] [49518.788723] [<ffffffffc166f987>] lod_it_next+0x27/0x90 [lod] [49518.857615] [<ffffffffc16e966f>] mdd_dir_page_build+0xaf/0x290 [mdd] [49518.934819] [<ffffffffc0da2e30>] dt_index_walk+0x1a0/0x430 [obdclass] [49519.013427] [<ffffffffc16eb24f>] mdd_readpage+0x25f/0x5a0 [mdd] [49519.085424] [<ffffffffc1584a1a>] mdt_readpage+0x63a/0x880 [mdt] [49519.157510] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [49519.239957] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [49519.331735] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [49519.406855] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [49519.465324] [<ffffffff9bd77c37>] ret_from_fork_nospec_end+0x0/0x39 [49519.540428] [<ffffffffffffffff>] 0xffffffffffffffff [49519.599956] LustreError: dumping log to /tmp/lustre-log.1567221770.16608 [49595.717452] Lustre: 33274:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567221840/real 1567221840] req@ffff8e50516c4b00 x1643303298690064/t0(0) o104->oak-MDT0000@10.8.27.2@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1567221847 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [49596.043147] Lustre: 33274:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [49611.319253] LNet: Service thread pid 16608 completed after 380.85s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [49672.722722] Lustre: 33258:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567221917/real 1567221917] req@ffff8e579d31fb00 x1643303298690736/t0(0) o104->oak-MDT0000@10.8.27.2@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1567221924 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [49673.048647] Lustre: 33258:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 20 previous similar messages [49694.166056] LustreError: 33258:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.8.27.2@o2ib6) failed to reply to blocking AST (req@ffff8e579d31fb00 x1643303298690736 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8e5328573a00/0x8223d6131b67c9da lrc: 4/0,0 mode: PR/PR res: [0x2000013b2:0x11:0x0].0x0 bits 0x13 rrc: 84 type: IBT flags: 0x60200400000020 nid: 10.8.27.2@o2ib6 remote: 0x4a298edc7d9c9867 expref: 36 pid: 33207 timeout: 4344453518 lvb_type: 0 [49694.168012] LustreError: 138-a: oak-MDT0000: A client on nid 10.8.27.2@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [49694.823600] LustreError: 33258:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) Skipped 1 previous similar message [49694.943318] LustreError: 138-a: oak-MDT0000: A client on nid 10.8.27.2@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [49748.690468] Lustre: oak-MDT0000: Client eb0aa1e6-740b-05dc-d421-610e2e0f23c5 (at 10.8.2.31@o2ib6) reconnecting [49748.810401] Lustre: Skipped 144 previous similar messages [49756.167986] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [49756.331388] LustreError: Skipped 1 previous similar message [49756.398073] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567221707, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5052f40600/0x8223d6131b54d27a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5a97a9630 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [49756.398299] LustreError: 40908:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6336270240) refcount nonzero (2) after lock cleanup; forcing cleanup. [49756.398301] LustreError: 40908:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6336270240) refcount = 3 [49756.398303] LustreError: 40908:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [49756.398311] LustreError: 40908:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5052f40600/0x8223d6131b54d27a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5a97a9630 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [49757.672667] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [49789.557746] LustreError: 16866:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [49789.664987] LustreError: 16866:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 44586 previous similar messages [50063.452971] LustreError: 40940:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e64bc79e840) refcount = 2 [50063.609093] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [50063.717364] Lustre: Skipped 94 previous similar messages [50131.156084] Lustre: DEBUG MARKER: Fri Aug 30 20:33:02 2019 [50368.746540] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [50368.909942] LustreError: Skipped 1 previous similar message [50368.976629] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567222320, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e680076f000/0x8223d6131d1df274 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5aa692fff expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [50368.976786] LustreError: 40990:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e622a4eecc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [50368.976787] LustreError: 40990:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [50368.976789] LustreError: 40990:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e622a4eecc0) refcount = 3 [50368.976791] LustreError: 40990:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [50368.976795] LustreError: 40990:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e680076f000/0x8223d6131d1df274 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5aa692fff expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [50370.376141] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [50389.572831] LustreError: 16602:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [50389.680082] LustreError: 16602:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 106863 previous similar messages [50679.574649] LustreError: 41037:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e63fe04dd40) refcount = 2 [50679.730794] LustreError: 41037:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [50679.830820] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [50679.939099] Lustre: Skipped 1 previous similar message [50985.322038] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [50985.485440] LustreError: Skipped 1 previous similar message [50985.552121] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567222936, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5ce3d7a600/0x8223d6131df952a9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ab430b83 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [50985.552283] LustreError: 41072:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5aa4280cc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [50985.552285] LustreError: 41072:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [50985.552287] LustreError: 41072:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5aa4280cc0) refcount = 3 [50985.552289] LustreError: 41072:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [50985.552294] LustreError: 41072:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5ce3d7a600/0x8223d6131df952a9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ab430b83 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [50985.552295] LustreError: 41072:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [50987.072386] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [50989.584602] LustreError: 16603:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [50989.691843] LustreError: 16603:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 132213 previous similar messages [51297.234056] LustreError: 41121:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e62581b0600) refcount = 2 [51297.390219] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [51297.498497] Lustre: Skipped 1 previous similar message [51589.596628] LustreError: 16883:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [51589.703898] LustreError: 16883:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 487156 previous similar messages [51602.642517] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [51602.805925] LustreError: Skipped 1 previous similar message [51602.872612] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567223553, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e54d21b3a00/0x8223d6131f44d25e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ac1d2f45 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [51602.872769] LustreError: 41174:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64c8cffd40) refcount nonzero (2) after lock cleanup; forcing cleanup. [51602.872770] LustreError: 41174:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [51602.872773] LustreError: 41174:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64c8cffd40) refcount = 3 [51602.872774] LustreError: 41174:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [51602.872779] LustreError: 41174:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e54d21b3a00/0x8223d6131f44d25e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ac1d2f45 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [51604.272170] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [51827.705216] Lustre: oak-MDT0000: haven't heard from client 8c863d09-8ea9-1a1e-532b-aa0f3bccf12d (at 10.9.104.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966e79000, cur 1567224079 expire 1567223929 last 1567223852 [51910.226548] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [51910.334882] Lustre: Skipped 1 previous similar message [52189.606994] LustreError: 37928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [52189.714249] LustreError: 37928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 486010 previous similar messages [52219.884095] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [52220.047523] LustreError: Skipped 1 previous similar message [52220.114235] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567224171, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f90a2ba00/0x8223d61329a58c2c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ad485154 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [52220.114424] LustreError: 41272:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e62f42bbc80) refcount nonzero (2) after lock cleanup; forcing cleanup. [52220.114427] LustreError: 41272:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e62f42bbc80) refcount = 3 [52220.114428] LustreError: 41272:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [52220.114433] LustreError: 41272:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f90a2ba00/0x8223d61329a58c2c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ad485154 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [52221.388867] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [52527.320112] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [52527.428394] Lustre: Skipped 2 previous similar messages [52789.618045] LustreError: 34428:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [52789.725628] LustreError: 34428:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 339547 previous similar messages [52836.941673] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [52837.105149] LustreError: Skipped 1 previous similar message [52837.171910] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567224788, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e69a8371000/0x8223d6133507e7ad lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ad84e0f6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [52837.172114] LustreError: 41350:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e653463f2c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [52837.172117] LustreError: 41350:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e653463f2c0) refcount = 3 [52837.172118] LustreError: 41350:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [52837.172123] LustreError: 41350:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e69a8371000/0x8223d6133507e7ad lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ad84e0f6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [52838.446520] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [53147.749795] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [53147.858077] Lustre: Skipped 1 previous similar message [53259.727383] Lustre: oak-MDT0000: haven't heard from client d2e5c963-f645-628f-36f5-9b4f969f041c (at 10.9.104.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966f92800, cur 1567225511 expire 1567225361 last 1567225284 [53278.795229] Lustre: 33247:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [53278.934736] Lustre: 33247:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 60 previous similar messages [53372.572093] Lustre: 33252:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [53372.711581] Lustre: 33252:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 38 previous similar messages [53389.715322] LustreError: 37937:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [53389.822571] LustreError: 37937:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 164482 previous similar messages [53457.189318] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [53457.352747] LustreError: Skipped 1 previous similar message [53457.419451] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567225408, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a87b5a00/0x8223d6134402a53d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5adce7c1d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [53457.419668] LustreError: 41424:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e611c7c6900) refcount nonzero (2) after lock cleanup; forcing cleanup. [53457.419670] LustreError: 41424:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e611c7c6900) refcount = 3 [53457.419672] LustreError: 41424:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [53457.419676] LustreError: 41424:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a87b5a00/0x8223d6134402a53d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5adce7c1d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [53458.694057] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [53765.398392] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [53765.506672] Lustre: Skipped 2 previous similar messages [53989.740434] LustreError: 37934:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [53989.847700] LustreError: 37934:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 71111 previous similar messages [54073.915873] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [54074.079283] LustreError: Skipped 1 previous similar message [54074.145981] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567226025, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a074f800/0x8223d6134debe71d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ae2256bc expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [54074.146149] LustreError: 41494:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5ef86d7440) refcount nonzero (2) after lock cleanup; forcing cleanup. [54074.146151] LustreError: 41494:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5ef86d7440) refcount = 3 [54074.146153] LustreError: 41494:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [54074.146158] LustreError: 41494:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a074f800/0x8223d6134debe71d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ae2256bc expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [54075.420618] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [54385.591881] LustreError: 41537:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5c6bf23b00) refcount = 2 [54385.748048] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [54385.856330] Lustre: Skipped 1 previous similar message [54589.797101] LustreError: 37939:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [54589.904344] LustreError: 37939:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 55323 previous similar messages [54692.868331] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [54693.031739] LustreError: Skipped 1 previous similar message [54693.098424] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567226644, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e609aaba200/0x8223d6135fb4312c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5af23082a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [54693.098547] LustreError: 41573:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6384f6ecc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [54693.098549] LustreError: 41573:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [54693.098551] LustreError: 41573:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6384f6ecc0) refcount = 3 [54693.098553] LustreError: 41573:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [54693.098557] LustreError: 41573:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e609aaba200/0x8223d6135fb4312c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5af23082a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [54694.497965] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [54834.146839] Lustre: oak-MDT0000: Client 0d40647a-311c-a59e-5918-f14604f1519a (at 10.8.2.22@o2ib6) reconnecting [54834.266563] Lustre: Skipped 50 previous similar messages [55001.729343] LustreError: 41612:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e65170a9740) refcount = 2 [55001.891707] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [55001.999981] Lustre: Skipped 3 previous similar messages [55132.823819] Lustre: 33249:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [55132.966916] Lustre: 33249:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages [55189.888546] LustreError: 16618:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [55189.995788] LustreError: 16618:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 88748 previous similar messages [55309.682901] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [55309.846300] LustreError: Skipped 1 previous similar message [55309.912996] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567227260, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a663dc00/0x8223d613694a988e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5afea25a5 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [55309.913239] LustreError: 41641:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4c3e6a0540) refcount nonzero (2) after lock cleanup; forcing cleanup. [55309.913241] LustreError: 41641:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [55309.913243] LustreError: 41641:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4c3e6a0540) refcount = 3 [55309.913244] LustreError: 41641:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [55309.913249] LustreError: 41641:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a663dc00/0x8223d613694a988e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5afea25a5 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [55311.312573] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [55433.486599] Lustre: 33095:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [55433.626094] Lustre: 33095:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages [55619.465837] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [55619.574116] Lustre: Skipped 1 previous similar message [55789.913059] LustreError: 34427:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [55790.020321] LustreError: 34427:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 127436 previous similar messages [55927.270430] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [55927.433847] LustreError: Skipped 1 previous similar message [55927.501192] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567227878, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5270b7a200/0x8223d61371a9331a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b0f71cca expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [55927.501373] LustreError: 42242:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e54e731c000) refcount nonzero (2) after lock cleanup; forcing cleanup. [55927.501376] LustreError: 42242:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e54e731c000) refcount = 3 [55927.501378] LustreError: 42242:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [55927.501384] LustreError: 42242:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5270b7a200/0x8223d61371a9331a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b0f71cca expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [55928.776395] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [56167.904238] Lustre: 33260:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [56168.043739] Lustre: 33260:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [56236.950403] LustreError: 42540:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5853a4d800) refcount = 2 [56237.106529] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [56237.214804] Lustre: Skipped 1 previous similar message [56355.590731] Lustre: oak-MDT0000: Client f57c2fff-051c-1ac2-8850-8c55e2caa91a (at 10.8.3.4@o2ib6) reconnecting [56389.927224] LustreError: 37928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [56390.034474] LustreError: 37928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 174078 previous similar messages [56417.825903] Lustre: oak-MDT0000: Client 06e283ef-2949-7d73-7834-58acfd699a0a (at 10.8.2.12@o2ib6) reconnecting [56543.801949] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [56543.965353] LustreError: Skipped 1 previous similar message [56544.032047] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567228495, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e56c2f20400/0x8223d61376c9a751 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b1d67223 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [56544.032210] LustreError: 43204:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5125a13200) refcount nonzero (2) after lock cleanup; forcing cleanup. [56544.032212] LustreError: 43204:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [56544.032214] LustreError: 43204:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5125a13200) refcount = 3 [56544.032216] LustreError: 43204:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [56544.032221] LustreError: 43204:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e56c2f20400/0x8223d61376c9a751 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b1d67223 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [56545.431545] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [56855.261011] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [56855.369302] Lustre: Skipped 5 previous similar messages [56989.977583] LustreError: 16857:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [56990.084830] LustreError: 16857:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 147022 previous similar messages [57055.600333] Lustre: 33240:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [57055.739845] Lustre: 33240:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages [57056.138579] Lustre: 33202:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [57056.278078] Lustre: 33202:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages [57163.944419] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [57164.107828] LustreError: Skipped 1 previous similar message [57164.174517] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567229115, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e659a340a00/0x8223d61381bd88f6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b2b26fd8 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [57164.174678] LustreError: 44181:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5afab869c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [57164.174681] LustreError: 44181:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5afab869c0) refcount = 3 [57164.174682] LustreError: 44181:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [57164.174686] LustreError: 44181:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e659a340a00/0x8223d61381bd88f6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b2b26fd8 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [57165.449131] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [57472.087315] LustreError: 44537:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5650339c80) refcount = 2 [57472.243440] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [57472.351716] Lustre: Skipped 1 previous similar message [57589.994279] LustreError: 37939:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [57590.101515] LustreError: 37939:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 151992 previous similar messages [57734.850534] Lustre: 33158:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [57734.990020] Lustre: 33158:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages [57778.410897] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [57778.574307] LustreError: Skipped 1 previous similar message [57778.641002] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567229729, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52cd374600/0x8223d6138b7abe0e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b390ed34 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [57778.641244] LustreError: 45069:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56306e0300) refcount nonzero (2) after lock cleanup; forcing cleanup. [57778.641245] LustreError: 45069:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [57778.641248] LustreError: 45069:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56306e0300) refcount = 3 [57778.641249] LustreError: 45069:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [57778.641254] LustreError: 45069:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52cd374600/0x8223d6138b7abe0e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b390ed34 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [57780.040588] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [58076.080111] Lustre: 33263:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [58076.219610] Lustre: 33263:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages [58085.704986] LustreError: 45542:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5397e0b980) refcount = 2 [58085.861109] LustreError: 45542:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [58085.961070] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [58086.069342] Lustre: Skipped 1 previous similar message [58190.094204] LustreError: 34424:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [58190.201447] LustreError: 34424:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 125618 previous similar messages [58301.441346] Lustre: oak-MDT0000: Client f57c2fff-051c-1ac2-8850-8c55e2caa91a (at 10.8.3.4@o2ib6) reconnecting [58301.560021] Lustre: Skipped 5 previous similar messages [58321.224126] Lustre: 33100:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [58321.363614] Lustre: 33100:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [58396.395369] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [58396.558791] LustreError: Skipped 1 previous similar message [58396.625491] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567230347, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4e4fbb6a00/0x8223d613977b09e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b462cf97 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [58396.625663] LustreError: 46020:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e548532f080) refcount nonzero (2) after lock cleanup; forcing cleanup. [58396.625665] LustreError: 46020:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [58396.625667] LustreError: 46020:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e548532f080) refcount = 3 [58396.625669] LustreError: 46020:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [58396.625674] LustreError: 46020:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4e4fbb6a00/0x8223d613977b09e3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b462cf97 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [58396.625675] LustreError: 46020:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [58398.146019] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [58705.123379] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [58705.231659] Lustre: Skipped 6 previous similar messages [58790.104460] LustreError: 37931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [58790.211699] LustreError: 37931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2062073 previous similar messages [58896.068465] Lustre: 33266:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [58896.207950] Lustre: 33266:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [59006.235614] Lustre: 33274:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [59006.375117] Lustre: 33274:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages [59014.382864] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [59014.546264] LustreError: Skipped 1 previous similar message [59014.612953] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567230965, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ad72a8e00/0x8223d613a92c84d3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b4b96502 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [59014.613155] LustreError: 46962:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4d7b6c4cc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [59014.613158] LustreError: 46962:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4d7b6c4cc0) refcount = 3 [59014.613160] LustreError: 46962:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [59014.613165] LustreError: 46962:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ad72a8e00/0x8223d613a92c84d3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b4b96502 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [59015.887574] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [59267.002453] Lustre: 33275:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [59267.141933] Lustre: 33275:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages [59324.934971] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [59325.043273] Lustre: Skipped 1 previous similar message [59325.528820] Lustre: 33254:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [59325.668300] Lustre: 33254:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages [59390.121751] LustreError: 33466:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [59390.228996] LustreError: 33466:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2918934 previous similar messages [59465.743186] Lustre: 16202:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [59550.151812] Lustre: 33079:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [59550.291304] Lustre: 33079:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [59634.973387] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [59635.136798] LustreError: Skipped 1 previous similar message [59635.203497] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567231586, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e503f5a5400/0x8223d613c0a4f443 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b500f0b3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [59635.203689] LustreError: 47876:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5053e34840) refcount nonzero (2) after lock cleanup; forcing cleanup. [59635.203692] LustreError: 47876:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5053e34840) refcount = 3 [59635.203694] LustreError: 47876:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [59635.203698] LustreError: 47876:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e503f5a5400/0x8223d613c0a4f443 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b500f0b3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [59636.478132] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [59732.970131] Lustre: 33199:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [59942.108381] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [59942.216691] Lustre: Skipped 1 previous similar message [59990.143502] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [59990.250745] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2561636 previous similar messages [60123.834378] Lustre: oak-MDT0000: haven't heard from client a4fdf60b-6e77-661f-215f-d5e1457a9a19 (at 10.12.4.80@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966571800, cur 1567232375 expire 1567232225 last 1567232148 [60249.476920] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [60249.640329] LustreError: Skipped 1 previous similar message [60249.707021] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567232200, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f68b5ac00/0x8223d613d8ab926a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b535330a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [60249.707213] LustreError: 48802:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5323245800) refcount nonzero (2) after lock cleanup; forcing cleanup. [60249.707216] LustreError: 48802:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5323245800) refcount = 3 [60249.707217] LustreError: 48802:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [60249.707222] LustreError: 48802:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f68b5ac00/0x8223d613d8ab926a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b535330a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [60250.981652] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [60313.593646] perf: interrupt took too long (4945 > 4943), lowering kernel.perf_event_max_sample_rate to 40000 [60560.941961] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [60561.050269] Lustre: Skipped 1 previous similar message [60590.156322] LustreError: 34446:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [60590.263583] LustreError: 34446:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2280394 previous similar messages [60648.357114] Lustre: 33228:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [60648.496599] Lustre: 33228:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages [60828.320894] Lustre: 16807:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [60828.460381] Lustre: 16807:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages [60868.590517] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [60868.753934] LustreError: Skipped 1 previous similar message [60868.820620] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567232819, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e514f3cde00/0x8223d613eafade27 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b572f78c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [60868.820806] LustreError: 49699:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4e38fdcd80) refcount nonzero (2) after lock cleanup; forcing cleanup. [60868.820809] LustreError: 49699:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4e38fdcd80) refcount = 3 [60868.820810] LustreError: 49699:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [60868.820815] LustreError: 49699:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e514f3cde00/0x8223d613eafade27 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b572f78c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [60870.095220] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [61002.126661] Lustre: 33280:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [61002.266144] Lustre: 33280:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages [61163.395881] Lustre: 33090:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [61163.535361] Lustre: 33090:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages [61177.765747] LustreError: 50182:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5443557080) refcount = 2 [61177.921883] LustreError: 50182:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [61178.021857] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [61178.130134] Lustre: Skipped 1 previous similar message [61190.171165] LustreError: 16876:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [61190.278406] LustreError: 16876:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 754109 previous similar messages [61486.662082] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [61486.825492] LustreError: Skipped 1 previous similar message [61486.892187] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567233437, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4eeb652800/0x8223d613f0a03c3f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b5af83f4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [61486.892474] LustreError: 50631:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e58a963f380) refcount nonzero (2) after lock cleanup; forcing cleanup. [61486.892476] LustreError: 50631:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [61486.892478] LustreError: 50631:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e58a963f380) refcount = 3 [61486.892479] LustreError: 50631:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [61486.892484] LustreError: 50631:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4eeb652800/0x8223d613f0a03c3f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b5af83f4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [61486.892485] LustreError: 50631:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [61488.412696] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [61790.181439] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [61790.288681] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1712428 previous similar messages [61793.693086] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [61793.801396] Lustre: Skipped 1 previous similar message [62100.643556] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [62100.806955] LustreError: Skipped 1 previous similar message [62100.873645] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567234051, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d84657200/0x8223d61401bfb0c7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b5f2cbb3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [62100.873966] LustreError: 51519:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56ef38d5c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [62100.873968] LustreError: 51519:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56ef38d5c0) refcount = 3 [62100.873970] LustreError: 51519:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [62100.873974] LustreError: 51519:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d84657200/0x8223d61401bfb0c7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b5f2cbb3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [62102.150069] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [62174.842262] Lustre: 33180:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [62174.981752] Lustre: 33180:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages [62390.230006] LustreError: 16861:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [62390.337257] LustreError: 16861:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1592044 previous similar messages [62408.751602] LustreError: 52008:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e50347fa540) refcount = 2 [62408.907774] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [62409.016066] Lustre: Skipped 1 previous similar message [62482.844982] Lustre: 33108:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [62482.984601] Lustre: 33108:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages [62718.659101] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [62718.822508] LustreError: Skipped 1 previous similar message [62718.889208] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567234669, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e684cbf0a00/0x8223d6141906060e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b6323053 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [62718.889445] LustreError: 52476:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e57528e3a40) refcount nonzero (2) after lock cleanup; forcing cleanup. [62718.889446] LustreError: 52476:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [62718.889448] LustreError: 52476:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e57528e3a40) refcount = 3 [62718.889450] LustreError: 52476:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [62718.889454] LustreError: 52476:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e684cbf0a00/0x8223d6141906060e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b6323053 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [62720.288761] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [62876.145897] Lustre: 16805:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [62876.285397] Lustre: 16805:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [62990.285240] LustreError: 37929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [62990.392483] LustreError: 37929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 176912 previous similar messages [63025.509175] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [63025.617467] Lustre: Skipped 1 previous similar message [63292.525435] Lustre: 33207:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [63292.665057] Lustre: 33207:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages [63331.567560] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [63331.730969] LustreError: Skipped 1 previous similar message [63331.797660] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567235282, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5966366c00/0x8223d6142e349acd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b6a982af expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [63331.797870] LustreError: 53319:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4d652be480) refcount nonzero (2) after lock cleanup; forcing cleanup. [63331.797872] LustreError: 53319:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4d652be480) refcount = 3 [63331.797874] LustreError: 53319:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [63331.797879] LustreError: 53319:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5966366c00/0x8223d6142e349acd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b6a982af expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [63333.073045] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [63590.322019] LustreError: 33399:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [63590.429262] LustreError: 33399:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1202838 previous similar messages [63604.887489] Lustre: oak-MDT0000: haven't heard from client 33d89aed-b8d5-40ab-a576-5ed10fd4f336 (at 10.8.27.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966573400, cur 1567235856 expire 1567235706 last 1567235629 [63639.327751] LustreError: 53792:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e54bc0652c0) refcount = 2 [63639.483879] LustreError: 53792:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [63639.583841] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [63639.692116] Lustre: Skipped 1 previous similar message [63948.122070] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [63948.285476] LustreError: Skipped 1 previous similar message [63948.352164] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567235899, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4edd249800/0x8223d61439c394db lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b730db2e expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [63948.352438] LustreError: 54126:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e52024f6480) refcount nonzero (2) after lock cleanup; forcing cleanup. [63948.352440] LustreError: 54126:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [63948.352442] LustreError: 54126:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e52024f6480) refcount = 3 [63948.352443] LustreError: 54126:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [63948.352448] LustreError: 54126:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4edd249800/0x8223d61439c394db lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b730db2e expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [63948.352449] LustreError: 54126:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [63949.877222] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [64190.631459] LustreError: 37937:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [64190.738704] LustreError: 37937:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 935348 previous similar messages [64255.480082] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [64255.588379] Lustre: Skipped 2 previous similar messages [64563.833512] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [64563.996913] LustreError: Skipped 1 previous similar message [64564.063599] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567236514, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51deadd600/0x8223d614426eae0a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b7a9b3df expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [64564.063864] LustreError: 55234:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e59b5f11e00) refcount nonzero (2) after lock cleanup; forcing cleanup. [64564.063866] LustreError: 55234:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e59b5f11e00) refcount = 3 [64564.063868] LustreError: 55234:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [64564.063873] LustreError: 55234:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51deadd600/0x8223d614426eae0a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b7a9b3df expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [64565.338211] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [64603.278432] Lustre: 33252:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [64603.417917] Lustre: 33252:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages [64665.017244] Lustre: 16601:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [64665.156720] Lustre: 16601:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages [64788.259789] Lustre: 33209:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [64788.399274] Lustre: 33209:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages [64790.900657] LustreError: 34427:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [64791.007896] LustreError: 34427:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 419109 previous similar messages [64871.972552] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [64872.080843] Lustre: Skipped 1 previous similar message [64874.415297] Lustre: 33285:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567237118/real 1567237118] req@ffff8e567065cb00 x1643304597686352/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567237125 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [64874.743030] Lustre: 33285:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [64895.860658] Lustre: 33285:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567237139/real 1567237139] req@ffff8e567065cb00 x1643304597686352/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567237146 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [64896.188387] Lustre: 33285:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [64928.358874] Lustre: 33249:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [64928.498358] Lustre: 33249:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 43 previous similar messages [64938.309312] Lustre: 33285:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567237182/real 1567237182] req@ffff8e567065cb00 x1643304597686352/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567237189 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [64938.637047] Lustre: 33285:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [64973.753840] LustreError: 33285:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.104.25@o2ib4) failed to reply to blocking AST (req@ffff8e567065cb00 x1643304597686352 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8e5dafd0be00/0x8223d61445e2f851 lrc: 4/0,0 mode: PR/PR res: [0x20002d4b3:0x7894:0x0].0x0 bits 0x13 rrc: 36 type: IBT flags: 0x60200400000020 nid: 10.9.104.25@o2ib4 remote: 0x85533c0a38790591 expref: 10557 pid: 33084 timeout: 4359732870 lvb_type: 0 [64974.271913] LustreError: 138-a: oak-MDT0000: A client on nid 10.9.104.25@o2ib4 was evicted due to a lock blocking callback time out: rc -110 [65177.237951] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [65177.401354] LustreError: Skipped 1 previous similar message [65177.468042] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567237128, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4db9f64000/0x8223d6144620738b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b7e524b9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [65177.468294] LustreError: 56077:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56cd37a900) refcount nonzero (2) after lock cleanup; forcing cleanup. [65177.468297] LustreError: 56077:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56cd37a900) refcount = 3 [65177.468298] LustreError: 56077:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [65177.468303] LustreError: 56077:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4db9f64000/0x8223d6144620738b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b7e524b9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [65178.742671] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [65198.610334] Lustre: 33270:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [65198.749816] Lustre: 33270:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 42 previous similar messages [65390.910252] LustreError: 34456:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [65391.017507] LustreError: 34456:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 396975 previous similar messages [65488.667093] LustreError: 56526:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e55f145f740) refcount = 2 [65488.823218] LustreError: 56526:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [65488.923188] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [65489.031465] Lustre: Skipped 2 previous similar messages [65742.923210] Lustre: 16592:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [65743.062707] Lustre: 16592:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 96 previous similar messages [65794.289478] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [65794.452881] LustreError: Skipped 1 previous similar message [65794.519567] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567237745, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5511db7600/0x8223d6144ca7eac7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b823eec6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [65794.519908] LustreError: 57016:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e49c90d8600) refcount nonzero (2) after lock cleanup; forcing cleanup. [65794.519910] LustreError: 57016:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [65794.519912] LustreError: 57016:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e49c90d8600) refcount = 3 [65794.519914] LustreError: 57016:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [65794.519921] LustreError: 57016:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5511db7600/0x8223d6144ca7eac7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b823eec6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [65794.519922] LustreError: 57016:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [65796.039865] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [65990.931427] LustreError: 16608:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [65991.038668] LustreError: 16608:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2272356 previous similar messages [66101.394421] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [66101.502711] Lustre: Skipped 1 previous similar message [66357.840777] Lustre: 33146:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [66357.980271] Lustre: 33146:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 38 previous similar messages [66407.461904] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [66407.625306] LustreError: Skipped 1 previous similar message [66407.691992] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567238358, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5417df4400/0x8223d6145a1c3f5a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b865e486 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [66407.692163] LustreError: 57923:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e58a9708f00) refcount nonzero (2) after lock cleanup; forcing cleanup. [66407.692165] LustreError: 57923:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e58a9708f00) refcount = 3 [66407.692167] LustreError: 57923:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [66407.692171] LustreError: 57923:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5417df4400/0x8223d6145a1c3f5a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b865e486 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [66408.966598] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [66590.940702] LustreError: 34442:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [66591.047946] LustreError: 34442:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 2340681 previous similar messages [66716.600848] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [66716.709140] Lustre: Skipped 1 previous similar message [66981.996828] Lustre: 33266:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [66982.136332] Lustre: 33266:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 66 previous similar messages [67023.935361] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [67024.098763] LustreError: Skipped 1 previous similar message [67024.165449] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567238974, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e60ae858000/0x8223d6146af7beff lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b8a584c9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [67024.165733] LustreError: 58838:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4dc5785800) refcount nonzero (2) after lock cleanup; forcing cleanup. [67024.165735] LustreError: 58838:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4dc5785800) refcount = 3 [67024.165737] LustreError: 58838:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [67024.165741] LustreError: 58838:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e60ae858000/0x8223d6146af7beff lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b8a584c9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [67025.440058] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [67190.952045] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [67191.059293] LustreError: 37924:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3360199 previous similar messages [67331.344529] LustreError: 59291:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56f9d08a80) refcount = 1 [67331.500658] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [67331.608931] Lustre: Skipped 1 previous similar message [67639.385831] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [67639.549238] LustreError: Skipped 1 previous similar message [67639.615932] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567239590, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ea5bdd200/0x8223d6147f172005 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b8e3944c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [67639.616125] LustreError: 59740:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56d7988cc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [67639.616127] LustreError: 59740:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [67639.616130] LustreError: 59740:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56d7988cc0) refcount = 3 [67639.616131] LustreError: 59740:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [67639.616136] LustreError: 59740:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ea5bdd200/0x8223d6147f172005 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b8e3944c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [67641.015480] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [67790.963245] LustreError: 37939:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [67791.070500] LustreError: 37939:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3556720 previous similar messages [67950.322902] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [67950.431185] Lustre: Skipped 1 previous similar message [67994.775476] Lustre: 33244:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [67994.914959] Lustre: 33244:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages [68257.337339] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [68257.500745] LustreError: Skipped 1 previous similar message [68257.567439] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567240208, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e58c5250e00/0x8223d614943d82c3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b91c3f40 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [68257.567669] LustreError: 60670:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e55b7296300) refcount nonzero (2) after lock cleanup; forcing cleanup. [68257.567671] LustreError: 60670:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e55b7296300) refcount = 3 [68257.567673] LustreError: 60670:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [68257.567677] LustreError: 60670:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e58c5250e00/0x8223d614943d82c3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b91c3f40 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [68258.842414] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [68390.974412] LustreError: 37935:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [68391.081657] LustreError: 37935:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3411867 previous similar messages [68568.861358] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [68568.969639] Lustre: Skipped 1 previous similar message [68600.420644] Lustre: 33192:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [68600.560129] Lustre: 33192:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 35 previous similar messages [68874.527835] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [68874.691245] LustreError: Skipped 1 previous similar message [68874.757939] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567240825, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f3d2dc400/0x8223d614a97e4317 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b94f9cc7 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [68874.758191] LustreError: 61552:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5085e02780) refcount nonzero (2) after lock cleanup; forcing cleanup. [68874.758195] LustreError: 61552:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5085e02780) refcount = 3 [68874.758196] LustreError: 61552:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [68874.758202] LustreError: 61552:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f3d2dc400/0x8223d614a97e4317 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b94f9cc7 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [68876.032579] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [68991.075548] LustreError: 37928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [68991.182794] LustreError: 37928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 3535154 previous similar messages [69182.229932] LustreError: 62041:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4eac2d3a40) refcount = 2 [69182.385979] LustreError: 62041:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [69182.485940] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [69182.594233] Lustre: Skipped 1 previous similar message [69470.076426] Lustre: 33097:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [69470.215909] Lustre: 33097:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 55 previous similar messages [69492.077332] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [69492.240747] LustreError: Skipped 1 previous similar message [69492.307447] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567241443, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e56aaa53c00/0x8223d614bde85290 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b978df9d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [69492.307638] LustreError: 62480:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4cdc2532c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [69492.307640] LustreError: 62480:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [69492.307643] LustreError: 62480:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4cdc2532c0) refcount = 3 [69492.307645] LustreError: 62480:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [69492.307654] LustreError: 62480:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e56aaa53c00/0x8223d614bde85290 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b978df9d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [69492.307656] LustreError: 62480:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [69493.829507] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [69497.065413] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567241741/real 1567241741] req@ffff8e4a13209200 x1643304950368864/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567241748 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [69497.393148] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [69511.509632] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567241755/real 1567241755] req@ffff8e4a13209200 x1643304950368864/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567241762 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [69511.837364] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 1 previous similar message [69532.953001] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567241776/real 1567241776] req@ffff8e4a13209200 x1643304950368864/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567241783 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [69533.280736] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [69575.397635] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567241819/real 1567241819] req@ffff8e4a13209200 x1643304950368864/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567241826 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [69575.725356] Lustre: 33286:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [69591.131246] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [69591.238492] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1339268 previous similar messages [69645.845712] LustreError: 33286:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.104.25@o2ib4) failed to reply to blocking AST (req@ffff8e4a13209200 x1643304950368864 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8e49f8639e00/0x8223d614c49c866d lrc: 4/0,0 mode: PR/PR res: [0x20002d4b3:0x789b:0x0].0x0 bits 0x13 rrc: 7 type: IBT flags: 0x60200400000020 nid: 10.9.104.25@o2ib4 remote: 0x143385de0db8cac7 expref: 6141 pid: 33209 timeout: 4364454890 lvb_type: 0 [69646.361707] LustreError: 138-a: oak-MDT0000: A client on nid 10.9.104.25@o2ib4 was evicted due to a lock blocking callback time out: rc -110 [69801.837348] LustreError: 62929:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4ef9284000) refcount = 2 [69801.993479] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [69802.101758] Lustre: Skipped 1 previous similar message [70108.020797] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [70108.184199] LustreError: Skipped 1 previous similar message [70108.250888] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567242059, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f10331400/0x8223d614c7693490 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b9a9b216 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [70108.251047] LustreError: 63392:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e50886b89c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [70108.251049] LustreError: 63392:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [70108.251051] LustreError: 63392:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e50886b89c0) refcount = 3 [70108.251052] LustreError: 63392:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [70108.251057] LustreError: 63392:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f10331400/0x8223d614c7693490 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b9a9b216 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [70109.651343] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [70138.220545] Lustre: 33081:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [70138.360025] Lustre: 33081:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 52 previous similar messages [70191.146303] LustreError: 16607:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [70191.253545] LustreError: 16607:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 264572 previous similar messages [70415.270883] LustreError: 63883:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e583a736e40) refcount = 2 [70415.427006] LustreError: 63883:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [70415.526971] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [70415.635261] Lustre: Skipped 2 previous similar messages [70724.938270] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [70725.101691] LustreError: Skipped 1 previous similar message [70725.168380] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567242675, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5271a52c00/0x8223d614c93ed8ca lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b9d7a6a0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [70725.168599] LustreError: 64323:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4e95ef1800) refcount nonzero (2) after lock cleanup; forcing cleanup. [70725.168601] LustreError: 64323:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [70725.168603] LustreError: 64323:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4e95ef1800) refcount = 3 [70725.168604] LustreError: 64323:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [70725.168609] LustreError: 64323:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5271a52c00/0x8223d614c93ed8ca lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b9d7a6a0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [70725.168610] LustreError: 64323:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [70726.688806] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [70791.162302] LustreError: 37940:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [70791.269542] LustreError: 37940:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 165574 previous similar messages [71036.708252] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [71036.816551] Lustre: Skipped 1 previous similar message [71084.203244] Lustre: 33072:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [71084.342740] Lustre: 33072:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 9 previous similar messages [71344.067749] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [71344.231152] LustreError: Skipped 1 previous similar message [71344.297844] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567243295, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5308dd0200/0x8223d614d29a80dd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5b9fffeb0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [71344.297975] LustreError: 65248:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e577e0f0b40) refcount nonzero (2) after lock cleanup; forcing cleanup. [71344.297978] LustreError: 65248:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e577e0f0b40) refcount = 3 [71344.297979] LustreError: 65248:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [71344.297984] LustreError: 65248:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5308dd0200/0x8223d614d29a80dd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5b9fffeb0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [71345.572529] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [71391.172512] LustreError: 34443:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [71391.279758] LustreError: 34443:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 870529 previous similar messages [71653.201698] LustreError: 65694:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e578a5a0180) refcount = 2 [71653.357862] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [71653.466176] Lustre: Skipped 1 previous similar message [71962.620257] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [71962.783667] LustreError: Skipped 1 previous similar message [71962.850358] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567243913, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4fcec07800/0x8223d614e9775695 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ba2a621c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [71962.850514] LustreError: 66134:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4a1471d740) refcount nonzero (2) after lock cleanup; forcing cleanup. [71962.850516] LustreError: 66134:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [71962.850520] LustreError: 66134:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4a1471d740) refcount = 3 [71962.850522] LustreError: 66134:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [71962.850527] LustreError: 66134:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4fcec07800/0x8223d614e9775695 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ba2a621c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [71964.249901] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [71971.816307] Lustre: 33242:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [71971.955908] Lustre: 33242:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages [71991.251219] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [71991.358455] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 519193 previous similar messages [72274.541524] LustreError: 66612:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e52f5722cc0) refcount = 2 [72274.697886] LustreError: 66612:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [72274.797875] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [72274.906155] Lustre: Skipped 1 previous similar message [72582.191801] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [72582.355201] LustreError: Skipped 1 previous similar message [72582.421888] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567244533, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5551f86a00/0x8223d614fd65d93d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ba518b18 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [72582.422057] LustreError: 67070:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e50547b46c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [72582.422058] LustreError: 67070:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [72582.422061] LustreError: 67070:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e50547b46c0) refcount = 3 [72582.422062] LustreError: 67070:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [72582.422067] LustreError: 67070:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5551f86a00/0x8223d614fd65d93d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ba518b18 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [72582.422068] LustreError: 67070:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [72583.942191] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [72591.280265] LustreError: 16184:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [72591.387505] LustreError: 16184:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 141970 previous similar messages [72599.104868] Lustre: 33199:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [72599.244352] Lustre: 33199:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 75 previous similar messages [72893.834863] LustreError: 67514:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4c7feec9c0) refcount = 2 [72893.990992] LustreError: 67514:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [72894.090963] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [72894.199250] Lustre: Skipped 1 previous similar message [73191.292378] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [73191.399618] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 589208 previous similar messages [73201.282345] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [73201.445756] LustreError: Skipped 1 previous similar message [73201.512453] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567245152, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e607f43c000/0x8223d61500791749 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ba779b2d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [73201.512645] LustreError: 67972:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5e07863080) refcount nonzero (2) after lock cleanup; forcing cleanup. [73201.512647] LustreError: 67972:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [73201.512650] LustreError: 67972:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5e07863080) refcount = 3 [73201.512651] LustreError: 67972:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [73201.512656] LustreError: 67972:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e607f43c000/0x8223d61500791749 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ba779b2d expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [73201.512657] LustreError: 67972:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [73203.032778] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [73414.385471] Lustre: 33154:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [73414.524957] Lustre: 33154:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 43 previous similar messages [73511.444508] LustreError: 68461:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e52d2899d40) refcount = 2 [73511.606880] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [73511.715172] Lustre: Skipped 1 previous similar message [73791.315623] LustreError: 16883:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [73791.422879] LustreError: 16883:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 397279 previous similar messages [73818.899903] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [73819.063311] LustreError: Skipped 1 previous similar message [73819.129997] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567245769, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e68e5a75a00/0x8223d6150a2296b1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ba9bf800 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [73819.130266] LustreError: 68904:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6510403140) refcount nonzero (2) after lock cleanup; forcing cleanup. [73819.130268] LustreError: 68904:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [73819.130270] LustreError: 68904:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6510403140) refcount = 3 [73819.130272] LustreError: 68904:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [73819.130276] LustreError: 68904:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e68e5a75a00/0x8223d6150a2296b1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ba9bf800 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [73820.529524] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [74014.613919] Lustre: 33272:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [74014.753398] Lustre: 33272:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages [74127.692031] LustreError: 69350:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e65306b63c0) refcount = 2 [74127.848159] LustreError: 69350:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [74127.948120] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [74128.056402] Lustre: Skipped 1 previous similar message [74391.329408] LustreError: 34451:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [74391.436653] LustreError: 34451:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 236662 previous similar messages [74435.060422] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [74435.223841] LustreError: Skipped 1 previous similar message [74435.290525] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567246386, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5d05538000/0x8223d6150b9a0dfd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5babf67b3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [74435.290775] LustreError: 69830:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6941e0f080) refcount nonzero (2) after lock cleanup; forcing cleanup. [74435.290777] LustreError: 69830:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [74435.290780] LustreError: 69830:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6941e0f080) refcount = 3 [74435.290781] LustreError: 69830:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [74435.290788] LustreError: 69830:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5d05538000/0x8223d6150b9a0dfd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5babf67b3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [74435.290789] LustreError: 69830:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [74436.810815] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [74617.592778] Lustre: 33154:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [74617.732264] Lustre: 33154:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages [74744.153313] LustreError: 70277:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e626be2cfc0) refcount = 2 [74744.315680] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [74744.423961] Lustre: Skipped 1 previous similar message [74991.356344] LustreError: 33402:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [74991.463583] LustreError: 33402:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 213516 previous similar messages [75050.607842] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [75050.771366] LustreError: Skipped 1 previous similar message [75050.838051] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567247001, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6590749600/0x8223d6150ce5454a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bae72a85 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [75050.838228] LustreError: 70720:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6363128540) refcount nonzero (2) after lock cleanup; forcing cleanup. [75050.838230] LustreError: 70720:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [75050.838232] LustreError: 70720:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6363128540) refcount = 3 [75050.838234] LustreError: 70720:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [75050.838238] LustreError: 70720:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6590749600/0x8223d6150ce5454a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bae72a85 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [75052.237571] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [75361.880887] LustreError: 71187:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e645b4e1740) refcount = 2 [75362.037014] LustreError: 71187:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [75362.136991] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [75362.245267] Lustre: Skipped 1 previous similar message [75591.367242] LustreError: 37941:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [75591.474496] LustreError: 37941:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 253178 previous similar messages [75649.599143] Lustre: 16601:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [75649.738636] Lustre: 16601:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages [75671.080266] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [75671.243681] LustreError: Skipped 1 previous similar message [75671.310362] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567247622, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e68127eb600/0x8223d6150e64a48b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bb0ebc1f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [75671.310617] LustreError: 71641:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5e13966180) refcount nonzero (2) after lock cleanup; forcing cleanup. [75671.310618] LustreError: 71641:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [75671.310620] LustreError: 71641:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5e13966180) refcount = 3 [75671.310622] LustreError: 71641:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [75671.310627] LustreError: 71641:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e68127eb600/0x8223d6150e64a48b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bb0ebc1f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [75671.310628] LustreError: 71641:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [75672.830646] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [75982.874270] LustreError: 72277:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e66c3f93c80) refcount = 1 [75983.030396] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [75983.138696] Lustre: Skipped 1 previous similar message [76191.381299] LustreError: 37941:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [76191.488545] LustreError: 37941:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 329171 previous similar messages [76288.413681] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [76288.577394] LustreError: Skipped 1 previous similar message [76288.644285] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567248239, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e61ca3e4600/0x8223d6150fed4886 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bb3558ab expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [76288.644529] LustreError: 72718:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64a0ed6cc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [76288.644531] LustreError: 72718:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [76288.644533] LustreError: 72718:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64a0ed6cc0) refcount = 3 [76288.644535] LustreError: 72718:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [76288.644542] LustreError: 72718:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e61ca3e4600/0x8223d6150fed4886 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bb3558ab expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [76290.043805] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [76373.552030] Lustre: 33187:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [76373.691528] Lustre: 33187:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages [76422.692717] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567248666/real 1567248666] req@ffff8e5421208c00 x1643305289635168/t0(0) o104->oak-MDT0000@10.9.104.27@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567248673 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [76423.020478] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [76437.137919] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567248681/real 1567248681] req@ffff8e5421208c00 x1643305289635168/t0(0) o104->oak-MDT0000@10.9.104.27@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567248688 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [76437.465644] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 1 previous similar message [76458.581277] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567248702/real 1567248702] req@ffff8e5421208c00 x1643305289635168/t0(0) o104->oak-MDT0000@10.9.104.27@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567248709 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [76458.909017] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [76501.027917] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567248744/real 1567248744] req@ffff8e5421208c00 x1643305289635168/t0(0) o104->oak-MDT0000@10.9.104.27@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567248751 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [76501.355657] Lustre: 33205:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [76522.472252] LustreError: 33205:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.104.27@o2ib4) failed to reply to blocking AST (req@ffff8e5421208c00 x1643305289635168 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8e4d662d7400/0x8223d6151147605f lrc: 4/0,0 mode: PR/PR res: [0x20002d4b3:0x7894:0x0].0x0 bits 0x13 rrc: 24 type: IBT flags: 0x60200400000020 nid: 10.9.104.27@o2ib4 remote: 0xba86005c106c2d15 expref: 6735 pid: 33101 timeout: 4371281411 lvb_type: 0 [76522.989262] LustreError: 138-a: oak-MDT0000: A client on nid 10.9.104.27@o2ib4 was evicted due to a lock blocking callback time out: rc -110 [76597.286621] LustreError: 73202:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5adbec2fc0) refcount = 2 [76597.442763] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [76597.551042] Lustre: Skipped 1 previous similar message [76791.392066] LustreError: 34448:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [76791.499305] LustreError: 34448:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 434587 previous similar messages [76907.227117] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [76907.390531] LustreError: Skipped 1 previous similar message [76907.457217] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567248858, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e663efea200/0x8223d615121f8559 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bb580f4e expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [76907.457474] LustreError: 73641:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5ac158d380) refcount nonzero (2) after lock cleanup; forcing cleanup. [76907.457476] LustreError: 73641:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [76907.457479] LustreError: 73641:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5ac158d380) refcount = 3 [76907.457480] LustreError: 73641:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [76907.457485] LustreError: 73641:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e663efea200/0x8223d615121f8559 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bb580f4e expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [76908.856746] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [76978.025233] Lustre: 33086:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [76978.164723] Lustre: 33086:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 47 previous similar messages [77215.412119] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [77215.520401] Lustre: Skipped 2 previous similar messages [77391.406006] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [77391.513267] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 448014 previous similar messages [77522.906532] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [77523.069941] LustreError: Skipped 1 previous similar message [77523.136635] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567249473, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e62da1c1c00/0x8223d615146c481e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bb7d2c85 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [77523.136857] LustreError: 74570:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5d2c084c00) refcount nonzero (2) after lock cleanup; forcing cleanup. [77523.136859] LustreError: 74570:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5d2c084c00) refcount = 3 [77523.136861] LustreError: 74570:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [77523.136866] LustreError: 74570:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e62da1c1c00/0x8223d615146c481e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bb7d2c85 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [77524.411300] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [77833.935538] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [77834.043819] Lustre: Skipped 1 previous similar message [77918.936352] Lustre: 33158:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [77919.075838] Lustre: 33158:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages [77991.416952] LustreError: 16614:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [77991.524191] LustreError: 16614:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 440723 previous similar messages [78140.330242] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [78140.493643] LustreError: Skipped 1 previous similar message [78140.560329] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567250091, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5eb67d2000/0x8223d61516a8cbee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bba93f3c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [78140.560636] LustreError: 75469:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5ceecbf980) refcount nonzero (2) after lock cleanup; forcing cleanup. [78140.560639] LustreError: 75469:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5ceecbf980) refcount = 3 [78140.560641] LustreError: 75469:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [78140.560645] LustreError: 75469:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5eb67d2000/0x8223d61516a8cbee lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bba93f3c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [78141.834940] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [78447.892387] LustreError: 75921:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64c7d38540) refcount = 2 [78448.048507] LustreError: 75921:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [78448.148481] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [78448.256809] Lustre: Skipped 1 previous similar message [78591.428420] LustreError: 16873:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [78591.535665] LustreError: 16873:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 452208 previous similar messages [78755.756768] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [78755.920174] LustreError: Skipped 1 previous similar message [78755.986856] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567250706, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e67932d2000/0x8223d61518fae009 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bbd30d9b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [78755.987118] LustreError: 76407:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6831719b00) refcount nonzero (2) after lock cleanup; forcing cleanup. [78755.987120] LustreError: 76407:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [78755.987122] LustreError: 76407:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6831719b00) refcount = 3 [78755.987123] LustreError: 76407:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [78755.987128] LustreError: 76407:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e67932d2000/0x8223d61518fae009 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bbd30d9b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [78755.987129] LustreError: 76407:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [78757.507153] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [78779.437106] Lustre: 33260:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [78779.576585] Lustre: 33260:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages [79067.262738] LustreError: 76840:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e5ecec9a840) refcount = 1 [79067.425121] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [79067.533400] Lustre: Skipped 1 previous similar message [79191.443394] LustreError: 16928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [79191.550637] LustreError: 16928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 480696 previous similar messages [79375.512368] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [79375.675778] LustreError: Skipped 1 previous similar message [79375.742487] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567251326, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c0023e400/0x8223d6151b6c5013 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bbfafba1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [79375.742754] LustreError: 77293:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64517f0540) refcount nonzero (2) after lock cleanup; forcing cleanup. [79375.742755] LustreError: 77293:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [79375.742758] LustreError: 77293:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64517f0540) refcount = 3 [79375.742759] LustreError: 77293:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [79375.742764] LustreError: 77293:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c0023e400/0x8223d6151b6c5013 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bbfafba1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [79377.142045] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [79424.077118] Lustre: 33136:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [79424.216604] Lustre: 33136:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [79682.599346] LustreError: 77758:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e584ba41080) refcount = 2 [79682.755475] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [79682.863752] Lustre: Skipped 1 previous similar message [79791.455225] LustreError: 34443:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [79791.562640] LustreError: 34443:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 413182 previous similar messages [79989.752815] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [79989.916220] LustreError: Skipped 1 previous similar message [79989.982911] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567251940, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6253564200/0x8223d6151d97bac1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bc1e66e6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [79989.983214] LustreError: 78218:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64fdf58900) refcount nonzero (2) after lock cleanup; forcing cleanup. [79989.983216] LustreError: 78218:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [79989.983219] LustreError: 78218:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64fdf58900) refcount = 3 [79989.983220] LustreError: 78218:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [79989.983225] LustreError: 78218:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6253564200/0x8223d6151d97bac1 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bc1e66e6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [79991.382443] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [80300.461884] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [80300.570163] Lustre: Skipped 1 previous similar message [80372.978082] Lustre: 16180:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [80373.117573] Lustre: 16180:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages [80391.467313] LustreError: 34461:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [80391.574553] LustreError: 34461:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 452046 previous similar messages [80608.899414] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [80609.062819] LustreError: Skipped 1 previous similar message [80609.129503] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567252559, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e69927f0e00/0x8223d6151fef699d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bc434651 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [80609.129818] LustreError: 79103:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e601213ee40) refcount nonzero (2) after lock cleanup; forcing cleanup. [80609.129825] LustreError: 79103:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e601213ee40) refcount = 3 [80609.129828] LustreError: 79103:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [80609.129840] LustreError: 79103:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e69927f0e00/0x8223d6151fef699d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bc434651 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [80610.404102] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [80915.998381] LustreError: 79605:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e5d1d282300) refcount = 2 [80916.160781] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [80916.269063] Lustre: Skipped 1 previous similar message [80991.500368] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [80991.607608] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 358318 previous similar messages [81021.661190] Lustre: 33277:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [81021.800672] Lustre: 33277:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 27 previous similar messages [81221.827846] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [81221.991253] LustreError: Skipped 1 previous similar message [81222.057940] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567253172, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6483de4000/0x8223d61521d1dc31 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bc6adcec expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [81222.058118] LustreError: 80048:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e63d367b140) refcount nonzero (2) after lock cleanup; forcing cleanup. [81222.058120] LustreError: 80048:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [81222.058122] LustreError: 80048:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e63d367b140) refcount = 3 [81222.058124] LustreError: 80048:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [81222.058128] LustreError: 80048:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6483de4000/0x8223d61521d1dc31 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bc6adcec expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [81223.457495] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [81532.347840] LustreError: 80486:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5c9f9ca900) refcount = 2 [81532.503974] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [81532.612255] Lustre: Skipped 1 previous similar message [81591.511709] LustreError: 37923:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [81591.618951] LustreError: 37923:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 393567 previous similar messages [81631.207110] Lustre: 33263:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [81631.346593] Lustre: 33263:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 64 previous similar messages [81840.866390] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [81841.029797] LustreError: Skipped 1 previous similar message [81841.096484] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567253791, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e662ca90a00/0x8223d615233709f0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bc948be0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [81841.096740] LustreError: 80930:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e60d321d5c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [81841.096742] LustreError: 80930:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [81841.096744] LustreError: 80930:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e60d321d5c0) refcount = 3 [81841.096746] LustreError: 80930:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [81841.096750] LustreError: 80930:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e662ca90a00/0x8223d615233709f0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bc948be0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [81842.496057] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [82149.924535] LustreError: 81411:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e60359f7bc0) refcount = 2 [82150.080659] LustreError: 81411:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [82150.180643] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [82150.288946] Lustre: Skipped 1 previous similar message [82191.523519] LustreError: 34424:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [82191.630758] LustreError: 34424:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 444224 previous similar messages [82325.496265] Lustre: 33152:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [82325.635745] Lustre: 33152:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 40 previous similar messages [82456.451953] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [82456.615369] LustreError: Skipped 1 previous similar message [82456.682064] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567254407, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e665bbf7c00/0x8223d61524fcefc0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bcbdb890 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [82456.682332] LustreError: 81854:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5f44bfa480) refcount nonzero (2) after lock cleanup; forcing cleanup. [82456.682334] LustreError: 81854:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [82456.682336] LustreError: 81854:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5f44bfa480) refcount = 3 [82456.682338] LustreError: 81854:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [82456.682342] LustreError: 81854:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e665bbf7c00/0x8223d61524fcefc0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bcbdb890 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [82456.682343] LustreError: 81854:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [82458.202334] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [82765.121906] LustreError: 82298:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5b2d9bc180) refcount = 2 [82765.278037] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [82765.386326] Lustre: Skipped 1 previous similar message [82791.537678] LustreError: 33400:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [82791.644919] LustreError: 33400:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 500679 previous similar messages [82968.390304] Lustre: 33200:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [82968.529796] Lustre: 33200:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 33 previous similar messages [83071.888395] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [83072.051802] LustreError: Skipped 1 previous similar message [83072.118485] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567255022, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6053bfb800/0x8223d61526df9a8c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bce5440a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [83072.118746] LustreError: 82780:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6382a73680) refcount nonzero (2) after lock cleanup; forcing cleanup. [83072.118750] LustreError: 82780:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [83072.118756] LustreError: 82780:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6382a73680) refcount = 3 [83072.118760] LustreError: 82780:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [83072.118772] LustreError: 82780:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6053bfb800/0x8223d61526df9a8c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bce5440a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [83073.517991] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [83381.150426] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [83381.258716] Lustre: Skipped 1 previous similar message [83391.548173] LustreError: 34459:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [83391.655412] LustreError: 34459:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 466240 previous similar messages [83638.598889] Lustre: 33149:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [83638.738370] Lustre: 33149:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 25 previous similar messages [83687.423894] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [83687.587310] LustreError: Skipped 1 previous similar message [83687.654013] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567255638, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a67c2400/0x8223d61528c6c56b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bd0a4b5a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [83687.654248] LustreError: 83660:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e635ac39440) refcount nonzero (2) after lock cleanup; forcing cleanup. [83687.654251] LustreError: 83660:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e635ac39440) refcount = 3 [83687.654253] LustreError: 83660:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [83687.654257] LustreError: 83660:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e60a67c2400/0x8223d61528c6c56b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bd0a4b5a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [83688.928627] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [83991.563349] LustreError: 16610:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [83991.670597] LustreError: 16610:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 530645 previous similar messages [83995.168888] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [83995.277188] Lustre: Skipped 1 previous similar message [84287.748779] Lustre: 33090:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [84287.888261] Lustre: 33090:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 20 previous similar messages [84303.658397] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [84303.821805] LustreError: Skipped 1 previous similar message [84303.888496] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567256254, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6441b7c000/0x8223d6152ac891be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bd2e099c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [84303.888789] LustreError: 84608:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e687da9f980) refcount nonzero (2) after lock cleanup; forcing cleanup. [84303.888791] LustreError: 84608:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e687da9f980) refcount = 3 [84303.888793] LustreError: 84608:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [84303.888797] LustreError: 84608:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6441b7c000/0x8223d6152ac891be lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bd2e099c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [84305.163148] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [84591.577000] LustreError: 37923:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [84591.684239] LustreError: 37923:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 599365 previous similar messages [84605.210228] Lustre: oak-MDT0000: haven't heard from client 7da8570e-6171-db3d-05b2-6eb6ba0b13e4 (at 10.9.104.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966844400, cur 1567256856 expire 1567256706 last 1567256629 [84612.745495] LustreError: 85049:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5fd8c79680) refcount = 2 [84612.901616] LustreError: 85049:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [84613.001592] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [84613.109863] Lustre: Skipped 1 previous similar message [84911.442877] Lustre: 33282:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [84911.582357] Lustre: 33282:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 19 previous similar messages [84922.863921] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [84923.027326] LustreError: Skipped 1 previous similar message [84923.094011] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567256873, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e699b6b6600/0x8223d6152d0707d0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bd4f3996 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [84923.094287] LustreError: 85493:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e644258dd40) refcount nonzero (2) after lock cleanup; forcing cleanup. [84923.094288] LustreError: 85493:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [84923.094291] LustreError: 85493:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e644258dd40) refcount = 3 [84923.094292] LustreError: 85493:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [84923.094299] LustreError: 85493:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e699b6b6600/0x8223d6152d0707d0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bd4f3996 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [84923.094300] LustreError: 85493:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [84924.614311] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [85174.219882] Lustre: oak-MDT0000: haven't heard from client 3ac94a66-98b7-ea22-67b7-b7e1fcbf26c7 (at 10.8.22.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966aee400, cur 1567257425 expire 1567257275 last 1567257198 [85191.587562] LustreError: 16928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [85191.694810] LustreError: 16928:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 525022 previous similar messages [85234.295953] LustreError: 85977:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e6577896780) refcount = 1 [85234.452081] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [85234.560355] Lustre: Skipped 1 previous similar message [85522.081640] Lustre: 33258:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [85522.221143] Lustre: 33258:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 24 previous similar messages [85540.624469] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [85540.787874] LustreError: Skipped 1 previous similar message [85540.854558] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567257491, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e64a5647e00/0x8223d6152f34edef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bd73c612 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [85540.854806] LustreError: 86421:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e65c2f1db00) refcount nonzero (2) after lock cleanup; forcing cleanup. [85540.854808] LustreError: 86421:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [85540.854810] LustreError: 86421:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e65c2f1db00) refcount = 3 [85540.854812] LustreError: 86421:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [85540.854817] LustreError: 86421:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e64a5647e00/0x8223d6152f34edef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bd73c612 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [85542.254143] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [85791.599428] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [85791.706667] LustreError: 16925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 599404 previous similar messages [85849.596595] LustreError: 86861:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e649e7ca900) refcount = 2 [85849.752716] LustreError: 86861:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [85849.852686] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [85849.960967] Lustre: Skipped 2 previous similar messages [86158.546010] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [86158.709414] LustreError: Skipped 1 previous similar message [86158.776101] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567258109, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e610197e600/0x8223d61531a874e6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bd994e02 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [86158.776353] LustreError: 87302:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6909b5d500) refcount nonzero (2) after lock cleanup; forcing cleanup. [86158.776355] LustreError: 87302:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [86158.776357] LustreError: 87302:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6909b5d500) refcount = 3 [86158.776359] LustreError: 87302:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [86158.776363] LustreError: 87302:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e610197e600/0x8223d61531a874e6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bd994e02 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [86158.776364] LustreError: 87302:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [86160.296395] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [86217.274622] Lustre: 33262:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [86217.414105] Lustre: 33262:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 43 previous similar messages [86391.609713] LustreError: 16604:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [86391.716953] LustreError: 16604:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 695734 previous similar messages [86466.928989] LustreError: 87786:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e61906d8900) refcount = 2 [86467.085151] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [86467.193443] Lustre: Skipped 1 previous similar message [86773.490496] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [86773.653901] LustreError: Skipped 1 previous similar message [86773.720582] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567258724, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5fd0297400/0x8223d6153448e07f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bdbc85bc expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [86773.720838] LustreError: 88238:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5c2e246480) refcount nonzero (2) after lock cleanup; forcing cleanup. [86773.720840] LustreError: 88238:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [86773.720842] LustreError: 88238:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5c2e246480) refcount = 3 [86773.720843] LustreError: 88238:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [86773.720848] LustreError: 88238:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5fd0297400/0x8223d6153448e07f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bdbc85bc expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [86775.120112] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [86991.620824] LustreError: 16606:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [86991.728069] LustreError: 16606:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1213090 previous similar messages [87041.115031] Lustre: 33285:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [87041.254515] Lustre: 33285:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 13 previous similar messages [87080.912456] LustreError: 88676:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e608de82300) refcount = 2 [87081.074827] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [87081.183106] Lustre: Skipped 2 previous similar messages [87095.249120] Lustre: oak-MDT0000: haven't heard from client 2873e9a0-8f23-cf06-7ceb-57d2292e1094 (at 10.9.110.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e68e067f400, cur 1567259346 expire 1567259196 last 1567259119 [87389.881004] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [87390.044410] LustreError: Skipped 1 previous similar message [87390.111093] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567259340, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6812a29000/0x8223d61537c3b077 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bde539d6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [87390.111258] LustreError: 89120:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64383315c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [87390.111260] LustreError: 89120:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [87390.111262] LustreError: 89120:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64383315c0) refcount = 3 [87390.111264] LustreError: 89120:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [87390.111268] LustreError: 89120:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6812a29000/0x8223d61537c3b077 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bde539d6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [87391.510629] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [87591.632833] LustreError: 34434:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [87591.740072] LustreError: 34434:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1052329 previous similar messages [87701.033960] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [87701.142240] Lustre: Skipped 1 previous similar message [88007.991528] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [88008.154928] LustreError: Skipped 1 previous similar message [88008.221612] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567259958, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5ff26aca00/0x8223d6153ad7a842 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5be0d2cf9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [88008.221882] LustreError: 90069:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e616565a180) refcount nonzero (2) after lock cleanup; forcing cleanup. [88008.221885] LustreError: 90069:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e616565a180) refcount = 3 [88008.221886] LustreError: 90069:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [88008.221891] LustreError: 90069:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5ff26aca00/0x8223d6153ad7a842 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5be0d2cf9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [88008.714546] Lustre: 33262:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [88008.714548] Lustre: 33262:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages [88009.751272] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [88191.644046] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [88191.751286] LustreError: 16931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1002995 previous similar messages [88318.879612] LustreError: 90511:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e5dba285680) refcount = 2 [88319.035739] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [88319.144018] Lustre: Skipped 2 previous similar messages [88629.054079] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [88629.217487] LustreError: Skipped 1 previous similar message [88629.284171] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567260579, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52038bfe00/0x8223d6153decbb1d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5be3e5c5c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [88629.284443] LustreError: 90982:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5d27bbea80) refcount nonzero (2) after lock cleanup; forcing cleanup. [88629.284445] LustreError: 90982:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [88629.284447] LustreError: 90982:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5d27bbea80) refcount = 3 [88629.284449] LustreError: 90982:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [88629.284453] LustreError: 90982:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52038bfe00/0x8223d6153decbb1d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5be3e5c5c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [88630.683707] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [88791.654708] LustreError: 16864:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [88791.762003] LustreError: 16864:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1193685 previous similar messages [88880.384286] Lustre: 33277:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [88880.523767] Lustre: 33277:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages [88940.002146] LustreError: 91426:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e6925746e40) refcount = 1 [88940.164517] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [88940.272800] Lustre: Skipped 1 previous similar message [89250.137660] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [89250.301061] LustreError: Skipped 1 previous similar message [89250.367746] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567261200, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e64f0e0f200/0x8223d61541ae79d2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5be70d0d6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [89250.367908] LustreError: 91890:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64aa0d3080) refcount nonzero (2) after lock cleanup; forcing cleanup. [89250.367910] LustreError: 91890:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [89250.367912] LustreError: 91890:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64aa0d3080) refcount = 3 [89250.367914] LustreError: 91890:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [89250.367918] LustreError: 91890:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e64f0e0f200/0x8223d61541ae79d2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5be70d0d6 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [89251.767302] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [89391.666087] LustreError: 33466:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [89391.773334] LustreError: 33466:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1212917 previous similar messages [89490.049299] Lustre: 33242:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [89490.189266] Lustre: 33242:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages [89514.285788] Lustre: oak-MDT0000: haven't heard from client 2501a83c-af98-24ff-c2fe-18ee904b722f (at 10.12.4.79@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966571000, cur 1567261765 expire 1567261615 last 1567261538 [89562.021816] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [89562.130384] Lustre: Skipped 1 previous similar message [89867.371235] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [89867.534643] LustreError: Skipped 1 previous similar message [89867.601327] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567261818, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4adbcd0000/0x8223d61545442adf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5be9dc8a3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [89867.601548] LustreError: 92816:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e65461cb680) refcount nonzero (2) after lock cleanup; forcing cleanup. [89867.601551] LustreError: 92816:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e65461cb680) refcount = 3 [89867.601553] LustreError: 92816:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [89867.601559] LustreError: 92816:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4adbcd0000/0x8223d61545442adf lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5be9dc8a3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [89868.875937] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [89935.292270] Lustre: oak-MDT0000: haven't heard from client 8f7ed388-b49c-c8ff-0acb-08f472eb9b7a (at 10.8.21.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966dcf800, cur 1567262186 expire 1567262036 last 1567261959 [89991.677690] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [89991.784948] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1016820 previous similar messages [90146.586487] Lustre: 33272:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [90146.725967] Lustre: 33272:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages [90174.505178] LustreError: 93263:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e6849a5f080) refcount = 2 [90174.667584] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [90174.775866] Lustre: Skipped 1 previous similar message [90482.348667] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [90482.512077] LustreError: Skipped 1 previous similar message [90482.578766] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567262433, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52f2e87200/0x8223d6154842b44e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bed4b025 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [90482.578991] LustreError: 93731:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64c6250600) refcount nonzero (2) after lock cleanup; forcing cleanup. [90482.578992] LustreError: 93731:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [90482.578995] LustreError: 93731:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64c6250600) refcount = 3 [90482.578996] LustreError: 93731:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [90482.579001] LustreError: 93731:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52f2e87200/0x8223d6154842b44e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bed4b025 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [90483.978319] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [90591.688891] LustreError: 16856:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [90591.796145] LustreError: 16856:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 833837 previous similar messages [90762.684995] Lustre: 33247:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [90762.824475] Lustre: 33247:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 10 previous similar messages [90778.305408] Lustre: oak-MDT0000: haven't heard from client 2ae464a0-18b5-757e-5be4-b617b0be0477 (at 10.9.109.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966577c00, cur 1567263029 expire 1567262879 last 1567262802 [90794.485756] LustreError: 94188:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e65d7c35800) refcount = 2 [90794.641883] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [90794.750160] Lustre: Skipped 2 previous similar messages [91102.583246] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [91102.746651] LustreError: Skipped 1 previous similar message [91102.813333] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567263053, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6816246800/0x8223d6154b1dbfe0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bf04cd1c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [91102.813532] LustreError: 94627:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6700efa180) refcount nonzero (2) after lock cleanup; forcing cleanup. [91102.813534] LustreError: 94627:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [91102.813537] LustreError: 94627:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6700efa180) refcount = 3 [91102.813539] LustreError: 94627:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [91102.813546] LustreError: 94627:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6816246800/0x8223d6154b1dbfe0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bf04cd1c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [91104.212882] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [91191.702359] LustreError: 34427:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [91191.809603] LustreError: 34427:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 968782 previous similar messages [91414.281301] LustreError: 95123:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5b1676f8c0) refcount = 2 [91414.437462] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [91414.545758] Lustre: Skipped 1 previous similar message [91608.119762] Lustre: 33202:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [91608.259144] Lustre: 33202:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [91719.758746] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [91719.922152] LustreError: Skipped 1 previous similar message [91719.988846] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567263670, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e681c7d6600/0x8223d6154e088fe0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bf393306 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [91719.989206] LustreError: 95578:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e640e6c0300) refcount nonzero (2) after lock cleanup; forcing cleanup. [91719.989209] LustreError: 95578:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [91719.989215] LustreError: 95578:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e640e6c0300) refcount = 3 [91719.989219] LustreError: 95578:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [91719.989231] LustreError: 95578:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e681c7d6600/0x8223d6154e088fe0 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bf393306 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [91721.388390] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [91791.712896] LustreError: 16853:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [91791.820144] LustreError: 16853:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 948341 previous similar messages [92029.381805] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [92029.490090] Lustre: Skipped 1 previous similar message [92335.855240] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [92336.018661] LustreError: Skipped 1 previous similar message [92336.085351] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567264286, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6989660200/0x8223d615516383ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bf71a8fc expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [92336.085585] LustreError: 96462:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4dfd28e9c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [92336.085587] LustreError: 96462:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4dfd28e9c0) refcount = 3 [92336.085589] LustreError: 96462:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [92336.085594] LustreError: 96462:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6989660200/0x8223d615516383ce lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bf71a8fc expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [92337.359968] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [92391.724114] LustreError: 37935:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [92391.831360] LustreError: 37935:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 976055 previous similar messages [92644.916278] LustreError: 96938:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e59282c0240) refcount = 2 [92645.072410] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [92645.180695] Lustre: Skipped 1 previous similar message [92953.169760] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [92953.333164] LustreError: Skipped 1 previous similar message [92953.399850] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567264903, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e631c51e400/0x8223d61554f18cfe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bfa46234 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [92953.400093] LustreError: 97376:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4baa383e00) refcount nonzero (2) after lock cleanup; forcing cleanup. [92953.400095] LustreError: 97376:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [92953.400097] LustreError: 97376:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4baa383e00) refcount = 3 [92953.400099] LustreError: 97376:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [92953.400103] LustreError: 97376:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e631c51e400/0x8223d61554f18cfe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bfa46234 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [92954.799774] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [92991.735988] LustreError: 16867:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [92991.843230] LustreError: 16867:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 972948 previous similar messages [93260.273726] LustreError: 97825:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4ff7801e00) refcount = 2 [93260.429885] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [93260.538167] Lustre: Skipped 1 previous similar message [93390.155063] Lustre: 33136:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [93390.294548] Lustre: 33136:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [93568.985264] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [93569.148671] LustreError: Skipped 1 previous similar message [93569.215357] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567265519, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f5c7fce00/0x8223d615589ec9ac lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5bfdab527 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [93569.215612] LustreError: 98282:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56337cc3c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [93569.215614] LustreError: 98282:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [93569.215617] LustreError: 98282:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56337cc3c0) refcount = 3 [93569.215620] LustreError: 98282:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [93569.215626] LustreError: 98282:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4f5c7fce00/0x8223d615589ec9ac lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5bfdab527 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [93570.614868] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [93591.747349] LustreError: 16604:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [93591.854591] LustreError: 16604:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 919014 previous similar messages [93646.313188] Lustre: 33183:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [93646.452670] Lustre: 33183:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [93878.474375] LustreError: 98774:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5bbebf58c0) refcount = 2 [93878.630532] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [93878.738830] Lustre: Skipped 2 previous similar messages [93939.933558] Lustre: 33228:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [93940.072944] Lustre: 33228:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [94185.434796] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [94185.598201] LustreError: Skipped 1 previous similar message [94185.664885] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567266136, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4baa2cfe00/0x8223d6155b731c6b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c009805e expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [94185.665104] LustreError: 99220:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e675b3723c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [94185.665105] LustreError: 99220:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [94185.665107] LustreError: 99220:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e675b3723c0) refcount = 3 [94185.665109] LustreError: 99220:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [94185.665114] LustreError: 99220:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4baa2cfe00/0x8223d6155b731c6b lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c009805e expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [94187.064407] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [94191.759285] LustreError: 33467:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [94191.866531] LustreError: 33467:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 680976 previous similar messages [94440.276179] Lustre: 33083:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [94440.415660] Lustre: 33083:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 16 previous similar messages [94496.478828] LustreError: 99660:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e62ee1a5440) refcount = 2 [94496.634954] LustreError: 99660:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [94496.734924] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [94496.843209] Lustre: Skipped 1 previous similar message [94791.769378] LustreError: 16613:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [94791.876615] LustreError: 16613:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 502975 previous similar messages [94804.694341] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [94804.857740] LustreError: Skipped 1 previous similar message [94804.924430] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567266755, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51e7468200/0x8223d6155dd018f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c03dd998 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [94804.924689] LustreError: 100144:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5d066d3c80) refcount nonzero (2) after lock cleanup; forcing cleanup. [94804.924690] LustreError: 100144:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [94804.924693] LustreError: 100144:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5d066d3c80) refcount = 3 [94804.924694] LustreError: 100144:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [94804.924699] LustreError: 100144:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51e7468200/0x8223d6155dd018f3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c03dd998 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [94804.924700] LustreError: 100144:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [94806.450956] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [95116.160540] LustreError: 100607:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e618a611140) refcount = 2 [95116.317726] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [95116.426009] Lustre: Skipped 1 previous similar message [95181.091346] Lustre: 16180:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [95181.230821] Lustre: 16180:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 6 previous similar messages [95391.780851] LustreError: 34431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [95391.888093] LustreError: 34431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 592074 previous similar messages [95422.656865] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [95422.820273] LustreError: Skipped 1 previous similar message [95422.886955] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567267373, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6575a26200/0x8223d6155ffa14ec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c06b242a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [95422.887247] LustreError: 101050:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e699bec8900) refcount nonzero (2) after lock cleanup; forcing cleanup. [95422.887248] LustreError: 101050:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [95422.887251] LustreError: 101050:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e699bec8900) refcount = 3 [95422.887252] LustreError: 101050:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [95422.887256] LustreError: 101050:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6575a26200/0x8223d6155ffa14ec lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c06b242a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [95424.291695] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [95732.437905] LustreError: 101541:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e580d27f8c0) refcount = 2 [95732.595088] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [95732.703375] Lustre: Skipped 1 previous similar message [95791.884071] Lustre: 33274:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [95792.023567] Lustre: 33274:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages [95991.793685] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [95991.900948] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 998282 previous similar messages [96038.073380] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [96038.236785] LustreError: Skipped 1 previous similar message [96038.303473] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567267988, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6438ecf600/0x8223d61562699bf9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c09d8931 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [96038.303752] LustreError: 101981:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5b46419980) refcount nonzero (2) after lock cleanup; forcing cleanup. [96038.303756] LustreError: 101981:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [96038.303762] LustreError: 101981:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5b46419980) refcount = 3 [96038.303766] LustreError: 101981:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [96038.303785] LustreError: 101981:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6438ecf600/0x8223d61562699bf9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c09d8931 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [96039.708211] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [96348.619631] LustreError: 102436:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e4f1a3d1e00) refcount = 2 [96348.783045] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [96348.891337] Lustre: Skipped 1 previous similar message [96374.366693] Lustre: 33250:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567268617/real 1567268617] req@ffff8e5eca666f00 x1643305304406192/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567268624 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [96374.694418] Lustre: 33250:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [96388.811938] Lustre: 33250:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567268632/real 1567268632] req@ffff8e5eca666f00 x1643305304406192/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567268639 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [96389.139672] Lustre: 33250:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 1 previous similar message [96410.255298] Lustre: 33250:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567268653/real 1567268653] req@ffff8e5eca666f00 x1643305304406192/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567268660 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [96410.583025] Lustre: 33250:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [96451.533956] Lustre: 33127:0:(client.c:2116:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1567268695/real 1567268695] req@ffff8e5d9370b600 x1643305304449712/t0(0) o104->oak-MDT0000@10.9.104.25@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1567268702 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [96451.861727] Lustre: 33127:0:(client.c:2116:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [96473.702278] LustreError: 33250:0:(ldlm_lockd.c:697:ldlm_handle_ast_error()) ### client (nid 10.9.104.25@o2ib4) failed to reply to blocking AST (req@ffff8e5eca666f00 x1643305304406192 status 0 rc -110), evict it ns: mdt-oak-MDT0000_UUID lock: ffff8e65e7338c00/0x8223d61563f485f7 lrc: 4/0,0 mode: PR/PR res: [0x20002f50e:0x2418:0x0].0x0 bits 0x13 rrc: 3 type: IBT flags: 0x60200400000020 nid: 10.9.104.25@o2ib4 remote: 0x32de1aa0abe2ac0f expref: 5090 pid: 33195 timeout: 4391232332 lvb_type: 0 [96474.218273] LustreError: 138-a: oak-MDT0000: A client on nid 10.9.104.25@o2ib4 was evicted due to a lock blocking callback time out: rc -110 [96507.335683] Lustre: 33283:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [96507.475160] Lustre: 33283:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 18 previous similar messages [96591.806100] LustreError: 34431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [96591.913337] LustreError: 34431:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 331244 previous similar messages [96658.760110] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [96658.923510] LustreError: Skipped 1 previous similar message [96658.990196] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567268609, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e613cedd400/0x8223d61564363f57 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c0db0727 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [96658.990490] LustreError: 102874:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e601fb7fb00) refcount nonzero (2) after lock cleanup; forcing cleanup. [96658.990492] LustreError: 102874:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [96658.990494] LustreError: 102874:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e601fb7fb00) refcount = 3 [96658.990496] LustreError: 102874:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [96658.990500] LustreError: 102874:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e613cedd400/0x8223d61564363f57 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c0db0727 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [96660.394922] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [96970.073285] LustreError: 103360:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e65280d4840) refcount = 2 [96970.230457] LustreError: 103360:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [96970.331465] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [96970.439737] Lustre: Skipped 2 previous similar messages [97178.204665] Lustre: 33072:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [97178.344159] Lustre: 33072:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 46 previous similar messages [97191.816716] LustreError: 34451:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [97191.923959] LustreError: 34451:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 438757 previous similar messages [97279.873745] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [97280.037147] LustreError: Skipped 1 previous similar message [97280.103840] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567269230, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e64c4faee00/0x8223d61565d8a090 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c11445a7 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [97280.104103] LustreError: 103806:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5fb2bbc180) refcount nonzero (2) after lock cleanup; forcing cleanup. [97280.104107] LustreError: 103806:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [97280.104113] LustreError: 103806:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5fb2bbc180) refcount = 3 [97280.104116] LustreError: 103806:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [97280.104128] LustreError: 103806:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e64c4faee00/0x8223d61565d8a090 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c11445a7 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [97280.104131] LustreError: 103806:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [97281.630411] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [97588.521832] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [97588.630117] Lustre: Skipped 1 previous similar message [97791.827910] LustreError: 33401:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [97791.935158] LustreError: 33401:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 866920 previous similar messages [97886.754992] Lustre: 33250:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [97886.894470] Lustre: 33250:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages [97897.784346] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [97897.947751] LustreError: Skipped 1 previous similar message [97898.014453] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567269848, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52c9b97200/0x8223d61567d4ad4d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c14fab59 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [97898.014886] LustreError: 104742:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64bce632c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [97898.014889] LustreError: 104742:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64bce632c0) refcount = 3 [97898.014890] LustreError: 104742:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [97898.014895] LustreError: 104742:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52c9b97200/0x8223d61567d4ad4d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c14fab59 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [97899.293222] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [98208.764336] LustreError: 105187:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e67f9269c80) refcount = 2 [98208.921532] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [98209.029922] Lustre: Skipped 1 previous similar message [98391.839024] LustreError: 33470:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [98391.946267] LustreError: 33470:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 865033 previous similar messages [98514.315816] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [98514.479224] LustreError: Skipped 1 previous similar message [98514.545915] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567270464, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6686275e00/0x8223d6156a254179 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c18d138b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [98514.546147] LustreError: 105630:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6447873bc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [98514.546149] LustreError: 105630:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [98514.546151] LustreError: 105630:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6447873bc0) refcount = 3 [98514.546153] LustreError: 105630:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [98514.546168] LustreError: 105630:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6686275e00/0x8223d6156a254179 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c18d138b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [98515.950660] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [98601.812421] Lustre: 33158:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [98601.951903] Lustre: 33158:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages [98650.427338] Lustre: oak-MDT0000: haven't heard from client 38335a85-7d3a-07df-a0ba-d7bce90f15b8 (at 10.8.27.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e69fa61dc00, cur 1567270901 expire 1567270751 last 1567270674 [98822.073648] LustreError: 106125:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e6366381a40) refcount = 2 [98822.230828] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [98822.339110] Lustre: Skipped 1 previous similar message [98991.851307] LustreError: 33400:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [98991.958547] LustreError: 33400:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 766554 previous similar messages [99131.183180] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [99131.346583] LustreError: Skipped 1 previous similar message [99131.413282] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567271081, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c49b37e00/0x8223d6156c48a465 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c1c298c9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [99131.413544] LustreError: 106577:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e625a00b2c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [99131.413545] LustreError: 106577:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [99131.413548] LustreError: 106577:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e625a00b2c0) refcount = 3 [99131.413549] LustreError: 106577:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [99131.413553] LustreError: 106577:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c49b37e00/0x8223d6156c48a465 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c1c298c9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [99132.818014] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [99238.678878] Lustre: 33267:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [99238.818369] Lustre: 33267:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 12 previous similar messages [99438.713118] LustreError: 107022:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e6343b27440) refcount = 2 [99438.870283] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [99438.978563] Lustre: Skipped 1 previous similar message [99591.862293] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [99591.969547] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 727549 previous similar messages [99747.453454] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [99747.616856] LustreError: Skipped 1 previous similar message [99747.683547] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567271698, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d46ccca00/0x8223d6156e4fbbb6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c1f53472 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [99747.683740] LustreError: 107468:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5cdcfc0000) refcount nonzero (2) after lock cleanup; forcing cleanup. [99747.683742] LustreError: 107468:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [99747.683744] LustreError: 107468:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5cdcfc0000) refcount = 3 [99747.683746] LustreError: 107468:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [99747.683750] LustreError: 107468:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d46ccca00/0x8223d6156e4fbbb6 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c1f53472 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [99749.088299] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [99850.606663] Lustre: 33248:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [99850.746143] Lustre: 33248:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 71 previous similar messages [100054.740480] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [100054.849803] Lustre: Skipped 1 previous similar message [100191.872352] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [100191.980651] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1010443 previous similar messages [100363.878932] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [100364.043378] LustreError: Skipped 1 previous similar message [100364.111104] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567272314, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e62d857e000/0x8223d61570b85c85 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c22a6f26 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [100364.111359] LustreError: 108416:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6085174780) refcount nonzero (2) after lock cleanup; forcing cleanup. [100364.111361] LustreError: 108416:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6085174780) refcount = 3 [100364.111364] LustreError: 108416:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [100364.111370] LustreError: 108416:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e62d857e000/0x8223d61570b85c85 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c22a6f26 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [100365.395099] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [100618.459714] Lustre: oak-MDT0000: haven't heard from client 212977aa-a235-be55-228e-e0f89236a3bd (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966c39000, cur 1567272869 expire 1567272719 last 1567272642 [100671.662964] LustreError: 108860:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e62fb971b00) refcount = 2 [100671.821205] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [100671.930526] Lustre: Skipped 1 previous similar message [100716.801007] Lustre: 33245:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [100716.941534] Lustre: 33245:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 22 previous similar messages [100791.883709] LustreError: 16611:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [100791.991989] LustreError: 16611:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1103241 previous similar messages [100977.653315] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [100977.817758] LustreError: Skipped 1 previous similar message [100977.885481] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567272928, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e67df720c00/0x8223d615734017e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c25aa101 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [100977.885717] LustreError: 109305:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e68f3ba2e40) refcount nonzero (2) after lock cleanup; forcing cleanup. [100977.885719] LustreError: 109305:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [100977.885721] LustreError: 109305:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e68f3ba2e40) refcount = 3 [100977.885723] LustreError: 109305:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [100977.885727] LustreError: 109305:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e67df720c00/0x8223d615734017e9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c25aa101 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [100979.296442] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [101288.352337] LustreError: 109794:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e63fb20a300) refcount = 2 [101288.510535] LustreError: 109794:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [101288.612581] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [101288.721915] Lustre: Skipped 3 previous similar messages [101328.540072] Lustre: 33242:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [101328.680611] Lustre: 33242:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 11 previous similar messages [101391.894640] LustreError: 34433:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [101392.002919] LustreError: 34433:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 875044 previous similar messages [101594.492730] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [101594.657172] LustreError: Skipped 1 previous similar message [101594.724904] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567273545, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e68286d8400/0x8223d61575a7f107 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c2915767 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [101594.725149] LustreError: 110242:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5d260fca80) refcount nonzero (2) after lock cleanup; forcing cleanup. [101594.725150] LustreError: 110242:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [101594.725153] LustreError: 110242:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5d260fca80) refcount = 3 [101594.725154] LustreError: 110242:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [101594.725158] LustreError: 110242:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e68286d8400/0x8223d61575a7f107 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c2915767 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [101594.725160] LustreError: 110242:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [101596.258703] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [101904.713830] LustreError: 110681:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e682a2ab680) refcount = 2 [101904.872032] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [101904.981348] Lustre: Skipped 1 previous similar message [101971.644254] Lustre: 33286:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [101971.784779] Lustre: 33286:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 15 previous similar messages [101991.905915] LustreError: 37926:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [101992.014198] LustreError: 37926:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 764448 previous similar messages [102213.745221] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [102213.909665] LustreError: Skipped 1 previous similar message [102213.977393] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567274164, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e50fda42800/0x8223d61577e759e2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c2ce92ba expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [102213.977618] LustreError: 111193:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e6249f5cc00) refcount nonzero (2) after lock cleanup; forcing cleanup. [102213.977620] LustreError: 111193:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [102213.977622] LustreError: 111193:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6249f5cc00) refcount = 3 [102213.977624] LustreError: 111193:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [102213.977628] LustreError: 111193:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e50fda42800/0x8223d61577e759e2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c2ce92ba expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [102215.388377] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [102524.406242] LustreError: 111641:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e6267060a80) refcount = 2 [102524.564448] LustreError: 111641:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [102524.666497] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [102524.775827] Lustre: Skipped 1 previous similar message [102591.918115] LustreError: 16185:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [102592.026408] LustreError: 16185:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 693973 previous similar messages [102831.042179] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [102831.206632] LustreError: Skipped 1 previous similar message [102831.274356] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567274781, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52c3a0a200/0x8223d6157a31ad30 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c30c77d4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [102831.274611] LustreError: 112089:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e63e75e4300) refcount nonzero (2) after lock cleanup; forcing cleanup. [102831.274613] LustreError: 112089:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [102831.274615] LustreError: 112089:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e63e75e4300) refcount = 3 [102831.274617] LustreError: 112089:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [102831.274621] LustreError: 112089:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52c3a0a200/0x8223d6157a31ad30 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c30c77d4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [102831.274622] LustreError: 112089:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [102832.808155] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [103141.683059] LustreError: 112536:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e64671ac300) refcount = 2 [103141.847509] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [103141.956830] Lustre: Skipped 1 previous similar message [103191.928807] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [103192.037089] LustreError: 34460:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 864652 previous similar messages [103230.497738] Lustre: oak-MDT0000: haven't heard from client d30e7b3e-610b-6bb5-2550-a52041fd8c18 (at 10.9.110.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e56a6728000, cur 1567275481 expire 1567275331 last 1567275254 [103448.455640] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [103448.620083] LustreError: Skipped 1 previous similar message [103448.687807] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567275398, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e61d2638800/0x8223d6157cf55323 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c3495a26 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [103448.687990] LustreError: 113018:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e64ab1f1d40) refcount nonzero (2) after lock cleanup; forcing cleanup. [103448.687991] LustreError: 113018:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [103448.687994] LustreError: 113018:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e64ab1f1d40) refcount = 3 [103448.687995] LustreError: 113018:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [103448.688001] LustreError: 113018:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e61d2638800/0x8223d6157cf55323 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c3495a26 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [103450.098777] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [103756.854690] LustreError: 113466:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e6807fb6a80) refcount = 2 [103757.012891] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [103757.122205] Lustre: Skipped 2 previous similar messages [103789.509595] Lustre: oak-MDT0000: haven't heard from client 89fc31e2-b743-d8e8-c992-ebfcf7f4ab59 (at 10.9.108.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966dc9400, cur 1567276040 expire 1567275890 last 1567275813 [103791.940886] LustreError: 37944:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [103792.049175] LustreError: 37944:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 819841 previous similar messages [104049.664204] Lustre: 33152:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [104049.804723] Lustre: 33152:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [104065.892202] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [104066.056659] LustreError: Skipped 1 previous similar message [104066.124392] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567276016, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e547b3c1600/0x8223d6157f897205 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c380a0f3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [104066.124579] LustreError: 113912:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5ac98078c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [104066.124581] LustreError: 113912:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [104066.124584] LustreError: 113912:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5ac98078c0) refcount = 3 [104066.124585] LustreError: 113912:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [104066.124589] LustreError: 113912:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e547b3c1600/0x8223d6157f897205 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c380a0f3 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [104067.535387] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [104377.382240] LustreError: 114393:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5e17346000) refcount = 2 [104377.540456] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [104377.649776] Lustre: Skipped 1 previous similar message [104391.951566] LustreError: 37943:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [104392.059844] LustreError: 37943:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 801857 previous similar messages [104686.342785] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [104686.507230] LustreError: Skipped 1 previous similar message [104686.574952] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567276636, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5b806b4e00/0x8223d61582029e2f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c3bc6564 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [104686.575152] LustreError: 114847:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e58e5b6b200) refcount nonzero (2) after lock cleanup; forcing cleanup. [104686.575154] LustreError: 114847:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [104686.575156] LustreError: 114847:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e58e5b6b200) refcount = 3 [104686.575158] LustreError: 114847:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [104686.575162] LustreError: 114847:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5b806b4e00/0x8223d61582029e2f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c3bc6564 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [104687.985905] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [104991.963574] LustreError: 33469:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [104992.071853] LustreError: 33469:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 674161 previous similar messages [104993.854797] LustreError: 115298:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e4f42bce3c0) refcount = 2 [104994.019247] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [104994.128566] Lustre: Skipped 1 previous similar message [105300.796386] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [105300.960833] LustreError: Skipped 1 previous similar message [105301.028560] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567277251, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e69d8a9c800/0x8223d615845f3ec2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c3f8938b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [105301.028825] LustreError: 115740:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e502bfed140) refcount nonzero (2) after lock cleanup; forcing cleanup. [105301.028827] LustreError: 115740:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [105301.028829] LustreError: 115740:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e502bfed140) refcount = 3 [105301.028831] LustreError: 115740:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [105301.028836] LustreError: 115740:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e69d8a9c800/0x8223d615845f3ec2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c3f8938b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [105302.439529] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [105591.974249] LustreError: 37940:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [105592.082531] LustreError: 37940:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 714625 previous similar messages [105610.451566] LustreError: 116214:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4d26b87e00) refcount = 2 [105610.609768] LustreError: 116214:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [105610.711844] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [105610.821186] Lustre: Skipped 1 previous similar message [105918.521051] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [105918.685496] LustreError: Skipped 1 previous similar message [105918.753222] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567277868, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e542669f200/0x8223d615870f84fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c42fa3ee expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [105918.753483] LustreError: 116691:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56f333f200) refcount nonzero (2) after lock cleanup; forcing cleanup. [105918.753484] LustreError: 116691:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [105918.753486] LustreError: 116691:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56f333f200) refcount = 3 [105918.753488] LustreError: 116691:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [105918.753492] LustreError: 116691:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e542669f200/0x8223d615870f84fd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c42fa3ee expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [105918.753493] LustreError: 116691:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [105920.287007] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [106101.528090] Lustre: 33097:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [106191.985532] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [106192.093711] LustreError: 34458:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 908166 previous similar messages [106227.493092] LustreError: 117141:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4eeb79b2c0) refcount = 2 [106227.651296] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [106227.760703] Lustre: Skipped 2 previous similar messages [106535.966613] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [106536.131054] LustreError: Skipped 1 previous similar message [106536.198801] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567278486, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52caa11200/0x8223d61589e31b9c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c464dd05 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [106536.199021] LustreError: 117616:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4c96e47bc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [106536.199023] LustreError: 117616:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [106536.199025] LustreError: 117616:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4c96e47bc0) refcount = 3 [106536.199027] LustreError: 117616:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [106536.199031] LustreError: 117616:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52caa11200/0x8223d61589e31b9c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c464dd05 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [106537.609891] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [106643.550340] Lustre: oak-MDT0000: haven't heard from client 1038e854-f679-3fca-f918-704432839b65 (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e4c84afec00, cur 1567278894 expire 1567278744 last 1567278667 [106643.811503] Lustre: Skipped 1 previous similar message [106791.996729] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [106792.105010] LustreError: 34449:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1096119 previous similar messages [106843.503822] LustreError: 118071:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5797320600) refcount = 2 [106843.662026] LustreError: 118071:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [106843.764076] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [106843.873407] Lustre: Skipped 2 previous similar messages [107150.825277] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [107150.989727] LustreError: Skipped 1 previous similar message [107151.057457] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567279101, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52f296c000/0x8223d6158d45c082 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c494af79 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [107151.057706] LustreError: 118526:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5232f629c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [107151.057707] LustreError: 118526:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [107151.057710] LustreError: 118526:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5232f629c0) refcount = 3 [107151.057711] LustreError: 118526:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [107151.057719] LustreError: 118526:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52f296c000/0x8223d6158d45c082 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c494af79 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [107151.057720] LustreError: 118526:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [107152.591254] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [107392.008094] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [107392.116369] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 959625 previous similar messages [107459.053329] LustreError: 118967:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e6221bdb740) refcount = 2 [107459.211541] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [107459.320857] Lustre: Skipped 2 previous similar messages [107766.502921] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [107766.667370] LustreError: Skipped 1 previous similar message [107766.735093] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567279716, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e62f7d1b600/0x8223d615904ee407 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c4c699f9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [107766.735274] LustreError: 119441:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5c626aa480) refcount nonzero (2) after lock cleanup; forcing cleanup. [107766.735276] LustreError: 119441:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [107766.735279] LustreError: 119441:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5c626aa480) refcount = 3 [107766.735280] LustreError: 119441:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [107766.735285] LustreError: 119441:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e62f7d1b600/0x8223d615904ee407 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c4c699f9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [107768.146070] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [107992.019825] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [107992.128110] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 967746 previous similar messages [108077.245101] LustreError: 119889:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e606bff5980) refcount = 2 [108077.403380] LustreError: 119889:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [108077.505450] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [108077.615104] Lustre: Skipped 1 previous similar message [108284.581550] Lustre: oak-MDT0000: haven't heard from client d957293f-c9cd-b582-9915-04bf3562572f (at 10.9.104.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6967389400, cur 1567280535 expire 1567280385 last 1567280308 [108284.843758] Lustre: Skipped 1 previous similar message [108387.668638] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [108387.833083] LustreError: Skipped 1 previous similar message [108387.900816] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567280338, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e60713d9000/0x8223d61593868808 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c4f6cfa1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [108387.901086] LustreError: 120352:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5e7ce5f980) refcount nonzero (2) after lock cleanup; forcing cleanup. [108387.901088] LustreError: 120352:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [108387.901091] LustreError: 120352:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5e7ce5f980) refcount = 3 [108387.901093] LustreError: 120352:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [108387.901099] LustreError: 120352:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e60713d9000/0x8223d61593868808 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c4f6cfa1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [108387.901101] LustreError: 120352:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [108389.434624] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [108548.582817] Lustre: oak-MDT0000: haven't heard from client e850fe5e-b7ea-a0f9-9588-62722291e0cb (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e56fd9ca400, cur 1567280799 expire 1567280649 last 1567280572 [108548.843989] Lustre: Skipped 1 previous similar message [108592.033921] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [108592.142203] LustreError: 34455:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 813434 previous similar messages [108651.120073] Lustre: 33252:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [108698.275757] LustreError: 120823:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e5f12647e00) refcount = 2 [108698.440210] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [108698.549539] Lustre: Skipped 1 previous similar message [108705.661171] Lustre: 16600:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [108705.801696] Lustre: 16600:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 4 previous similar messages [109004.910276] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [109005.074718] LustreError: Skipped 1 previous similar message [109005.142446] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567280955, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5d8d182000/0x8223d6159686d34c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c5294ab2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [109005.142685] LustreError: 121267:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e60d15a2fc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [109005.142689] LustreError: 121267:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [109005.142695] LustreError: 121267:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e60d15a2fc0) refcount = 3 [109005.142699] LustreError: 121267:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [109005.142711] LustreError: 121267:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5d8d182000/0x8223d6159686d34c lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c5294ab2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [109006.557587] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [109077.580011] Lustre: 16600:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [109077.720444] Lustre: 16600:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [109192.044794] LustreError: 16864:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [109192.153086] LustreError: 16864:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 959123 previous similar messages [109314.976336] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [109315.085662] Lustre: Skipped 2 previous similar messages [109381.595557] Lustre: oak-MDT0000: haven't heard from client cf4559b8-dd96-6de7-f6a7-ca53742922de (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e649fc5d800, cur 1567281632 expire 1567281482 last 1567281405 [109575.108904] Lustre: 33144:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [109575.249426] Lustre: 33144:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages [109622.937968] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [109623.102429] LustreError: Skipped 1 previous similar message [109623.170157] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567281573, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6487b30c00/0x8223d61599df1c35 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c55266b4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [109623.170398] LustreError: 122196:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5cdc2e4000) refcount nonzero (2) after lock cleanup; forcing cleanup. [109623.170405] LustreError: 122196:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5cdc2e4000) refcount = 3 [109623.170409] LustreError: 122196:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [109623.170428] LustreError: 122196:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6487b30c00/0x8223d61599df1c35 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c55266b4 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [109623.793923] Lustre: 33194:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [109623.793927] Lustre: 33194:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 2 previous similar messages [109624.710228] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [109792.055849] LustreError: 16856:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [109792.164153] LustreError: 16856:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1186289 previous similar messages [109931.893084] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [109932.002403] Lustre: Skipped 2 previous similar messages [109940.784302] Lustre: 33222:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [109940.925210] Lustre: 33222:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 7 previous similar messages [110240.485607] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [110240.650053] LustreError: Skipped 1 previous similar message [110240.717777] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567282190, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e69b9777c00/0x8223d6159dae689e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c5889d61 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [110240.718014] LustreError: 123001:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e51d8690cc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [110240.718016] LustreError: 123001:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e51d8690cc0) refcount = 3 [110240.718018] LustreError: 123001:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [110240.718022] LustreError: 123001:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e69b9777c00/0x8223d6159dae689e lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c5889d61 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [110242.001773] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [110287.607569] Lustre: oak-MDT0000: haven't heard from client c36db0b8-5979-6787-26c6-83972fc34cd4 (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e58777bac00, cur 1567282538 expire 1567282388 last 1567282311 [110392.067913] LustreError: 16608:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [110392.176193] LustreError: 16608:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1303187 previous similar messages [110548.725632] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [110548.834956] Lustre: Skipped 3 previous similar messages [110858.751282] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [110858.915793] LustreError: Skipped 1 previous similar message [110858.983591] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567282809, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e560a858a00/0x8223d615a2232fbe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c5bf50da expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [110858.983859] LustreError: 123637:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e543a81c000) refcount nonzero (2) after lock cleanup; forcing cleanup. [110858.983862] LustreError: 123637:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e543a81c000) refcount = 3 [110858.983863] LustreError: 123637:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [110858.983870] LustreError: 123637:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e560a858a00/0x8223d615a2232fbe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c5bf50da expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [110860.267675] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [110992.078890] LustreError: 34430:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [110992.187171] LustreError: 34430:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1700000 previous similar messages [111058.619454] Lustre: oak-MDT0000: haven't heard from client 62e72472-4470-5dca-9a23-3e1da58a8fbe (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e58cd4f3000, cur 1567283309 expire 1567283159 last 1567283082 [111170.421389] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [111170.530710] Lustre: Skipped 3 previous similar messages [111480.515014] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [111480.679457] LustreError: Skipped 1 previous similar message [111480.747183] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567283430, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6953afa000/0x8223d615a6f67c87 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c5f62657 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [111480.747398] LustreError: 124238:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4e7d6ac240) refcount nonzero (2) after lock cleanup; forcing cleanup. [111480.747400] LustreError: 124238:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4e7d6ac240) refcount = 3 [111480.747402] LustreError: 124238:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [111480.747406] LustreError: 124238:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6953afa000/0x8223d615a6f67c87 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c5f62657 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [111482.031165] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [111592.089920] LustreError: 37944:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [111592.198206] LustreError: 37944:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1459743 previous similar messages [111793.284118] LustreError: 124658:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e523fff35c0) refcount = 2 [111793.448641] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [111793.558021] Lustre: Skipped 2 previous similar messages [112101.129746] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [112101.294194] LustreError: Skipped 1 previous similar message [112101.361913] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567284051, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e59cf716200/0x8223d615aa91b1b8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c62da436 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [112101.362146] LustreError: 124953:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e571ceaa540) refcount nonzero (2) after lock cleanup; forcing cleanup. [112101.362147] LustreError: 124953:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [112101.362150] LustreError: 124953:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e571ceaa540) refcount = 3 [112101.362151] LustreError: 124953:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [112101.362155] LustreError: 124953:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e59cf716200/0x8223d615aa91b1b8 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c62da436 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [112102.773500] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [112192.102234] LustreError: 34440:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [112192.210513] LustreError: 34440:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 968051 previous similar messages [112380.640142] Lustre: oak-MDT0000: haven't heard from client d1a64367-af7a-429d-24d6-f9a928117795 (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e58cd04a000, cur 1567284631 expire 1567284481 last 1567284404 [112413.130852] LustreError: 125278:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e4f70f5afc0) refcount = 2 [112413.295303] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [112413.404653] Lustre: Skipped 1 previous similar message [112723.387425] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [112723.552173] LustreError: Skipped 1 previous similar message [112723.620266] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567284673, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e61219d9a00/0x8223d615aead5708 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c65e9ba0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [112723.620402] LustreError: 125770:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e51ad6af500) refcount nonzero (2) after lock cleanup; forcing cleanup. [112723.620403] LustreError: 125770:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [112723.620405] LustreError: 125770:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e51ad6af500) refcount = 3 [112723.620407] LustreError: 125770:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [112723.620412] LustreError: 125770:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e61219d9a00/0x8223d615aead5708 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c65e9ba0 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [112725.033077] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [112795.565813] LustreError: 16609:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [112795.674100] LustreError: 16609:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 894032 previous similar messages [113034.745509] LustreError: 126099:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e58e56395c0) refcount = 2 [113034.903718] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [113035.013038] Lustre: Skipped 1 previous similar message [113347.319165] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [113347.483691] LustreError: Skipped 1 previous similar message [113347.551424] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567285297, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e529e6f2800/0x8223d615b2ffd126 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c696b347 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [113347.551661] LustreError: 126418:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4dcafc2300) refcount nonzero (2) after lock cleanup; forcing cleanup. [113347.551663] LustreError: 126418:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [113347.551665] LustreError: 126418:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4dcafc2300) refcount = 3 [113347.551666] LustreError: 126418:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [113347.551671] LustreError: 126418:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e529e6f2800/0x8223d615b2ffd126 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c696b347 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [113348.963719] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [113395.576555] LustreError: 33401:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [113395.684840] LustreError: 33401:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 834451 previous similar messages [113655.678166] LustreError: 127115:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e55b4534480) refcount = 2 [113655.842699] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [113655.952040] Lustre: Skipped 2 previous similar messages [113805.167226] LNet: Service thread pid 16602 was inactive for 200.07s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [113805.371186] Pid: 16602, comm: mdt_rdpg00_002 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 [113805.499224] Call Trace: [113805.529502] [<ffffffff9b945d03>] get_request+0x243/0x7d0 [113805.595248] [<ffffffff9b948bce>] blk_queue_bio+0xfe/0x400 [113805.662040] [<ffffffff9b946e17>] generic_make_request+0x147/0x380 [113805.737143] [<ffffffff9b9470c0>] submit_bio+0x70/0x150 [113805.800812] [<ffffffff9b87b627>] _submit_bh+0x127/0x160 [113805.865520] [<ffffffff9b87c079>] ll_rw_block+0xa9/0xb0 [113805.929185] [<ffffffffc1380163>] ldiskfs_bread+0x63/0xc0 [ldiskfs] [113806.005345] [<ffffffffc133c0ca>] __ldiskfs_read_dirblock+0x4a/0x400 [ldiskfs] [113806.092943] [<ffffffffc133c4c0>] htree_dirblock_to_tree+0x40/0x190 [ldiskfs] [113806.179492] [<ffffffffc133e6a1>] ldiskfs_htree_fill_tree+0x201/0x2f0 [ldiskfs] [113806.268123] [<ffffffffc1331b0c>] ldiskfs_readdir+0x61c/0x850 [ldiskfs] [113806.348424] [<ffffffffc144fece>] osd_ldiskfs_it_fill+0xbe/0x260 [osd_ldiskfs] [113806.436021] [<ffffffffc145023e>] osd_it_ea_next+0xce/0x180 [osd_ldiskfs] [113806.518402] [<ffffffffc166f987>] lod_it_next+0x27/0x90 [lod] [113806.588314] [<ffffffffc16e966f>] mdd_dir_page_build+0xaf/0x290 [mdd] [113806.666551] [<ffffffffc0da2e30>] dt_index_walk+0x1a0/0x430 [obdclass] [113806.745853] [<ffffffffc16eb24f>] mdd_readpage+0x25f/0x5a0 [mdd] [113806.818884] [<ffffffffc1584a1a>] mdt_readpage+0x63a/0x880 [mdt] [113806.891917] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [113806.975406] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [113807.068390] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [113807.144650] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [113807.204172] [<ffffffff9bd77c37>] ret_from_fork_nospec_end+0x0/0x39 [113807.280323] [<ffffffffffffffff>] 0xffffffffffffffff [113807.340880] LustreError: dumping log to /tmp/lustre-log.1567286057.16602 [113817.763075] LNet: Service thread pid 16602 completed after 212.67s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [113964.388723] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [113964.553172] LustreError: Skipped 1 previous similar message [113964.620907] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567285914, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e52f5a6da00/0x8223d615bc982d7f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c6c968ff expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [113964.621095] LustreError: 127410:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5531f1c9c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [113964.621097] LustreError: 127410:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [113964.621100] LustreError: 127410:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5531f1c9c0) refcount = 3 [113964.621101] LustreError: 127410:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [113964.621106] LustreError: 127410:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e52f5a6da00/0x8223d615bc982d7f lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c6c968ff expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [113966.032385] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [113995.587909] LustreError: 34434:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [113995.696192] LustreError: 34434:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 784554 previous similar messages [114276.333816] LustreError: 127788:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5794e1d380) refcount = 2 [114276.492048] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [114276.601386] Lustre: Skipped 1 previous similar message [114587.720473] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [114587.884914] LustreError: Skipped 1 previous similar message [114587.952642] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567286538, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5513bb2800/0x8223d615c5848ed3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c6f9d4a1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [114587.952896] LustreError: 128467:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4d5f2f5380) refcount nonzero (2) after lock cleanup; forcing cleanup. [114587.952898] LustreError: 128467:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [114587.952900] LustreError: 128467:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4d5f2f5380) refcount = 3 [114587.952902] LustreError: 128467:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [114587.952906] LustreError: 128467:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5513bb2800/0x8223d615c5848ed3 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c6f9d4a1 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [114589.364523] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [114595.599775] LustreError: 37935:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [114595.708056] LustreError: 37935:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 693391 previous similar messages [114628.684630] Lustre: oak-MDT0000: haven't heard from client 9f87f658-a9f0-5939-172d-f78aee53c9f5 (at 10.9.106.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966bdb000, cur 1567286879 expire 1567286729 last 1567286652 [114898.907655] LustreError: 130042:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4a11149080) refcount = 2 [114899.065863] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [114899.175182] Lustre: Skipped 1 previous similar message [115195.613306] LustreError: 16859:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [115195.721607] LustreError: 16859:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 616994 previous similar messages [115205.004188] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [115205.168634] LustreError: Skipped 1 previous similar message [115205.236360] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567287155, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4cd5614600/0x8223d615ce169aea lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c7278394 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [115205.236612] LustreError: 130557:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e50a8e2f740) refcount nonzero (2) after lock cleanup; forcing cleanup. [115205.236613] LustreError: 130557:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [115205.236616] LustreError: 130557:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e50a8e2f740) refcount = 3 [115205.236617] LustreError: 130557:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [115205.236622] LustreError: 130557:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4cd5614600/0x8223d615ce169aea lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c7278394 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [115206.647330] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [115260.376527] Lustre: 16601:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [115260.517049] Lustre: 16601:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 3 previous similar messages [115514.585306] LustreError: 131014:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e565de9e0c0) refcount = 2 [115514.743520] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [115514.852866] Lustre: Skipped 2 previous similar messages [115626.492434] Lustre: 16591:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [115795.624201] LustreError: 33399:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [115795.732483] LustreError: 33399:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 595635 previous similar messages [115824.935920] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [115825.100361] LustreError: Skipped 1 previous similar message [115825.168082] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567287775, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e540ecf5200/0x8223d615d60b6314 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c7580b59 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [115825.168278] LustreError: 131481:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e53b6eaf980) refcount nonzero (2) after lock cleanup; forcing cleanup. [115825.168279] LustreError: 131481:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [115825.168281] LustreError: 131481:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e53b6eaf980) refcount = 3 [115825.168283] LustreError: 131481:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [115825.168287] LustreError: 131481:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e540ecf5200/0x8223d615d60b6314 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c7580b59 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [115826.579075] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [116136.020034] LustreError: 131983:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e552a25af00) refcount = 2 [116136.184477] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [116136.293790] Lustre: Skipped 1 previous similar message [116395.634916] LustreError: 16930:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [116395.743200] LustreError: 16930:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 453963 previous similar messages [116443.954582] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [116444.119033] LustreError: Skipped 1 previous similar message [116444.186773] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567288394, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e520152d400/0x8223d615de1d6ee2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c78402e2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [116444.187009] LustreError: 132465:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e58eafc6300) refcount nonzero (2) after lock cleanup; forcing cleanup. [116444.187011] LustreError: 132465:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [116444.187013] LustreError: 132465:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e58eafc6300) refcount = 3 [116444.187015] LustreError: 132465:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [116444.187019] LustreError: 132465:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e520152d400/0x8223d615de1d6ee2 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c78402e2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [116445.597725] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [116697.100831] Lustre: 33242:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [116755.652647] LustreError: 132956:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4d7d6e4780) refcount = 2 [116755.810859] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [116755.920176] Lustre: Skipped 2 previous similar messages [116995.646387] LustreError: 16873:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [116995.754680] LustreError: 16873:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 516941 previous similar messages [117063.722258] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [117063.886701] LustreError: Skipped 1 previous similar message [117063.954427] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567289014, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5198f6a600/0x8223d615e56d20bd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c7b08c4c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [117063.954596] LustreError: 133444:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4c9aec8180) refcount nonzero (2) after lock cleanup; forcing cleanup. [117063.954597] LustreError: 133444:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [117063.954600] LustreError: 133444:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4c9aec8180) refcount = 3 [117063.954601] LustreError: 133444:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [117063.954606] LustreError: 133444:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5198f6a600/0x8223d615e56d20bd lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c7b08c4c expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [117065.365392] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [117103.473592] Lustre: 33252:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [117109.986635] Lustre: 33110:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [117269.181732] Lustre: 33279:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [117372.496406] LustreError: 133938:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e58e576b680) refcount = 2 [117372.654609] LustreError: 133938:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [117372.756659] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [117372.865977] Lustre: Skipped 1 previous similar message [117595.668224] LustreError: 16927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [117595.776508] LustreError: 16927:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 472887 previous similar messages [117680.269904] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [117680.434351] LustreError: Skipped 1 previous similar message [117680.502081] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567289630, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e53d65dfc00/0x8223d615eb9ed203 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c7d7bff9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [117680.502271] LustreError: 134406:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e53c798d680) refcount nonzero (2) after lock cleanup; forcing cleanup. [117680.502272] LustreError: 134406:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [117680.502275] LustreError: 134406:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e53c798d680) refcount = 3 [117680.502277] LustreError: 134406:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [117680.502281] LustreError: 134406:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e53d65dfc00/0x8223d615eb9ed203 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c7d7bff9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [117680.502283] LustreError: 134406:0:(ldlm_resource.c:1706:ldlm_resource_dump()) Skipped 1 previous similar message [117682.035867] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [117720.723620] Lustre: oak-MDT0000: haven't heard from client 15f2cfcc-d2b5-f80c-c0f0-76e61e80b6fa (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e4db08fa400, cur 1567289971 expire 1567289821 last 1567289744 [117974.586577] Lustre: 33267:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [117989.546974] LustreError: 134883:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e5a943ec600) refcount = 2 [117989.705215] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [117989.814553] Lustre: Skipped 1 previous similar message [118195.679818] LustreError: 34423:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [118195.788096] LustreError: 34423:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 526604 previous similar messages [118272.732203] Lustre: oak-MDT0000: haven't heard from client 6d305697-0b9c-a038-c0ba-da077161a88b (at 10.9.107.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e69669e6c00, cur 1567290523 expire 1567290373 last 1567290296 [118300.058557] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [118300.223018] LustreError: Skipped 1 previous similar message [118300.290939] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567290250, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e56e1f5ea00/0x8223d615f37ba5c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c8070bc9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [118300.291152] LustreError: 135368:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e566abc60c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [118300.291154] LustreError: 135368:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [118300.291157] LustreError: 135368:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e566abc60c0) refcount = 3 [118300.291159] LustreError: 135368:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [118300.291165] LustreError: 135368:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e56e1f5ea00/0x8223d615f37ba5c4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c8070bc9 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [118301.703225] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [118609.880741] LustreError: 135856:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4a19163680) refcount = 2 [118610.038948] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [118610.148266] Lustre: Skipped 2 previous similar messages [118730.195825] Lustre: 33147:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [118730.336348] Lustre: 33147:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [118795.691546] LustreError: 37931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [118795.799824] LustreError: 37931:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 470189 previous similar messages [118873.741508] Lustre: oak-MDT0000: haven't heard from client 7c831bbf-3206-31c9-cd65-1244715aa297 (at 10.8.17.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6966b21800, cur 1567291124 expire 1567290974 last 1567290897 [118920.037225] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [118920.201679] LustreError: Skipped 1 previous similar message [118920.269403] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567290870, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4b98b0ea00/0x8223d615fb621414 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c83b0bca expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [118920.269672] LustreError: 136319:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56aee14480) refcount nonzero (2) after lock cleanup; forcing cleanup. [118920.269673] LustreError: 136319:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [118920.269675] LustreError: 136319:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56aee14480) refcount = 3 [118920.269677] LustreError: 136319:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [118920.269682] LustreError: 136319:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4b98b0ea00/0x8223d615fb621414 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c83b0bca expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [118921.680399] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [119083.055978] Lustre: 33190:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [119205.356999] Lustre: 33247:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [119224.265970] Lustre: oak-MDT0000: Connection restored to 7c831bbf-3206-31c9-cd65-1244715aa297 (at 10.8.17.19@o2ib6) [119224.390891] Lustre: Skipped 1 previous similar message [119228.663266] LustreError: 136831:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4fc32d8b40) refcount = 2 [119382.762882] Lustre: 33257:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [119382.903405] Lustre: 33257:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 8 previous similar messages [119395.702890] LustreError: 37922:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [119395.811178] LustreError: 37922:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 510515 previous similar messages [119539.396879] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [119539.561324] LustreError: Skipped 1 previous similar message [119539.629070] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567291489, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5266a63e00/0x8223d61603021654 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c86c252a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [119539.629227] LustreError: 137296:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e522c560840) refcount nonzero (2) after lock cleanup; forcing cleanup. [119539.629229] LustreError: 137296:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [119539.629232] LustreError: 137296:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e522c560840) refcount = 3 [119539.629234] LustreError: 137296:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [119539.629243] LustreError: 137296:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5266a63e00/0x8223d61603021654 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c86c252a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [119541.040563] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [119849.660988] LustreError: 137773:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e547d7d0240) refcount = 2 [119849.825448] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [119849.934768] Lustre: Skipped 2 previous similar messages [119995.721071] LustreError: 16875:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [119995.829356] LustreError: 16875:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 525810 previous similar messages [120018.759508] Lustre: oak-MDT0000: haven't heard from client 015161d6-6490-10b3-7932-67f15bdea024 (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e5802397c00, cur 1567292269 expire 1567292119 last 1567292042 [120159.178608] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [120159.343053] LustreError: Skipped 1 previous similar message [120159.410793] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567292109, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ec37cd600/0x8223d6160adc9038 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c8983e7f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [120159.411022] LustreError: 138287:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e57e174fc80) refcount nonzero (2) after lock cleanup; forcing cleanup. [120159.411023] LustreError: 138287:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [120159.411026] LustreError: 138287:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e57e174fc80) refcount = 3 [120159.411027] LustreError: 138287:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [120159.411031] LustreError: 138287:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ec37cd600/0x8223d6160adc9038 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c8983e7f expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [120160.821807] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [120467.693556] LustreError: 138772:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e4b8077c480) refcount = 2 [120467.858014] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [120467.967363] Lustre: Skipped 1 previous similar message [120595.732648] LustreError: 37933:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [120595.840939] LustreError: 37933:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 550424 previous similar messages [120777.557216] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [120777.721660] LustreError: Skipped 1 previous similar message [120777.789388] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567292727, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e51ede91c00/0x8223d61612b68fd4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c8c0e277 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [120777.789526] LustreError: 139229:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5461e4cfc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [120777.789527] LustreError: 139229:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [120777.789530] LustreError: 139229:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5461e4cfc0) refcount = 3 [120777.789531] LustreError: 139229:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [120777.789536] LustreError: 139229:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e51ede91c00/0x8223d61612b68fd4 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c8c0e277 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [120779.200371] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [121086.861480] LustreError: 139734:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e53b13ce9c0) refcount = 2 [121087.019737] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [121087.129077] Lustre: Skipped 2 previous similar messages [121195.744658] LustreError: 34429:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [121195.852954] LustreError: 34429:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 669429 previous similar messages [121396.951011] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [121397.115510] LustreError: Skipped 1 previous similar message [121397.183243] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567293347, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5258e06a00/0x8223d6162000d71d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c8e34b25 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [121397.183389] LustreError: 140212:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e5722752480) refcount nonzero (2) after lock cleanup; forcing cleanup. [121397.183390] LustreError: 140212:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [121397.183392] LustreError: 140212:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e5722752480) refcount = 3 [121397.183394] LustreError: 140212:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [121397.183399] LustreError: 140212:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5258e06a00/0x8223d6162000d71d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c8e34b25 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [121398.595433] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [121423.659390] Lustre: 33239:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [121423.799916] Lustre: 33239:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 1 previous similar message [121504.782862] Lustre: oak-MDT0000: haven't heard from client b7c7216d-78f2-3ddc-3a6b-d6cc98bbfea7 (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e693fe16c00, cur 1567293755 expire 1567293605 last 1567293528 [121708.731080] LustreError: 140684:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e4eb1825e00) refcount = 1 [121708.895533] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [121709.004855] Lustre: Skipped 2 previous similar messages [121795.755895] LustreError: 16863:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [121795.864180] LustreError: 16863:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 660110 previous similar messages [121977.802280] Lustre: 33141:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [122015.453943] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [122015.618386] LustreError: Skipped 1 previous similar message [122015.686119] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567293965, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e5479697000/0x8223d61630a37b5a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c9077802 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [122015.686298] LustreError: 141185:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e547b787140) refcount nonzero (2) after lock cleanup; forcing cleanup. [122015.686299] LustreError: 141185:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [122015.686301] LustreError: 141185:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e547b787140) refcount = 3 [122015.686303] LustreError: 141185:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [122015.686312] LustreError: 141185:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e5479697000/0x8223d61630a37b5a lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c9077802 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [122017.097138] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [122327.061052] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [122327.170388] Lustre: Skipped 2 previous similar messages [122354.418255] Lustre: 33081:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [122395.766970] LustreError: 16929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [122395.875255] LustreError: 16929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 607680 previous similar messages [122637.524671] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [122637.689122] LustreError: Skipped 1 previous similar message [122637.756845] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567294587, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ee2395e00/0x8223d61643f5cbe7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c92760fb expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [122637.757002] LustreError: 142133:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e56e63b55c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [122637.757005] LustreError: 142133:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56e63b55c0) refcount = 3 [122637.757006] LustreError: 142133:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [122637.757011] LustreError: 142133:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4ee2395e00/0x8223d61643f5cbe7 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c92760fb expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [122639.040892] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [122804.334729] Lustre: 33241:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [122865.459508] Lustre: 33217:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [122945.957747] LustreError: 142628:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x0:0x0].0x0 (ffff8e4ecbb166c0) refcount = 2 [122946.115953] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [122946.225284] Lustre: Skipped 1 previous similar message [122995.783178] LustreError: 16929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [122995.891458] LustreError: 16929:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 514448 previous similar messages [123038.907602] Lustre: 33113:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [123039.048124] Lustre: 33113:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [123086.509318] Lustre: 33209:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [123095.808308] Lustre: oak-MDT0000: haven't heard from client 49e7019c-0024-8d3e-1b1a-a40d03a24b7e (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e50cf0a2800, cur 1567295346 expire 1567295196 last 1567295119 [123256.234440] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [123256.398901] LustreError: Skipped 1 previous similar message [123256.466896] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567295206, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e579d39b000/0x8223d616577849ef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c94ccf61 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [123256.467130] LustreError: 143115:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e532b7cbc80) refcount nonzero (2) after lock cleanup; forcing cleanup. [123256.467131] LustreError: 143115:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [123256.467134] LustreError: 143115:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e532b7cbc80) refcount = 3 [123256.467135] LustreError: 143115:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [123256.467139] LustreError: 143115:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e579d39b000/0x8223d616577849ef lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c94ccf61 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [123257.877975] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [123354.416743] Lustre: 33200:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [123354.557278] Lustre: 33200:0:(mdd_device.c:1577:mdd_changelog_clear()) Skipped 5 previous similar messages [123569.214664] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [123569.324014] Lustre: Skipped 2 previous similar messages [123586.015246] Lustre: 33079:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [123595.793900] LustreError: 37922:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [123595.902195] LustreError: 37922:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 423354 previous similar messages [123877.383229] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [123877.547686] LustreError: Skipped 1 previous similar message [123877.615431] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567295827, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e6675e55e00/0x8223d6166e8e2afe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c9716672 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [123877.615681] LustreError: 144100:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e65056080c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [123877.615684] LustreError: 144100:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e65056080c0) refcount = 3 [123877.615686] LustreError: 144100:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [123877.615691] LustreError: 144100:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e6675e55e00/0x8223d6166e8e2afe lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c9716672 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [123878.899478] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [124188.915465] LustreError: 144585:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e4d2fb3fa40) refcount = 2 [124189.075461] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [124189.184788] Lustre: Skipped 1 previous similar message [124195.805996] LustreError: 16864:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [124195.914290] LustreError: 16864:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 376050 previous similar messages [124279.667283] Lustre: 33135:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [124496.662979] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [124496.827419] LustreError: Skipped 1 previous similar message [124496.895147] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567296446, 301s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e50da3f2c00/0x8223d6168d1ff749 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c99ba094 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [124496.895345] LustreError: 145040:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e578561b200) refcount nonzero (2) after lock cleanup; forcing cleanup. [124496.895347] LustreError: 145040:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [124496.895350] LustreError: 145040:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e578561b200) refcount = 3 [124496.895351] LustreError: 145040:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [124496.895356] LustreError: 145040:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e50da3f2c00/0x8223d6168d1ff749 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c99ba094 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [124498.307012] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [124759.577313] Lustre: 33156:0:(mdd_device.c:1577:mdd_changelog_clear()) oak-MDD0000: Failure to clear the changelog for user 1: -22 [124795.816721] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [124795.925010] LustreError: 16855:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 319553 previous similar messages [124808.248072] LustreError: 145540:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x5:0x0].0x0 (ffff8e588430b380) refcount = 2 [124808.406621] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [124808.515959] Lustre: Skipped 1 previous similar message [125106.844015] Lustre: oak-MDT0000: haven't heard from client 6ef377d1-cb3e-af63-590d-fa17472f630c (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e586fadc800, cur 1567297357 expire 1567297207 last 1567297130 [125116.072734] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [125116.237302] LustreError: Skipped 1 previous similar message [125116.305044] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567297066, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c00743800/0x8223d616a81d51d9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c9c624ad expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [125116.305333] LustreError: 146024:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e55d45a1a40) refcount nonzero (2) after lock cleanup; forcing cleanup. [125116.305335] LustreError: 146024:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [125116.305337] LustreError: 146024:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e55d45a1a40) refcount = 3 [125116.305339] LustreError: 146024:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [125116.305343] LustreError: 146024:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4c00743800/0x8223d616a81d51d9 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c9c624ad expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [125117.717219] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [125182.841179] Lustre: oak-MDT0000: haven't heard from client 36297b76-7ade-3648-c959-be2228e3736c (at 10.8.27.13@o2ib6) in 165 seconds. I think it's dead, and I am evicting it. exp ffff8e6966c74000, cur 1567297433 expire 1567297283 last 1567297268 [125183.102375] Lustre: Skipped 1 previous similar message [125398.412387] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [125398.520673] LustreError: 16612:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 941270 previous similar messages [125423.610899] LustreError: 146318:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x736d61726170:0x3:0x0].0x0 (ffff8e53f6344480) refcount = 2 [125423.775563] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [125423.884880] Lustre: Skipped 1 previous similar message [125476.887212] Lustre: oak-MDT0000: Client bf016d2e-4371-4096-8391-627cd1e360e7 (at 10.8.30.17@o2ib6) reconnecting [125477.009019] Lustre: Skipped 1 previous similar message [125574.785809] Lustre: oak-MDT0000: Client beb1e17f-f7bc-ab10-f1ae-b868d9512bd5 (at 10.9.110.4@o2ib4) reconnecting [125741.925605] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [125742.090046] LustreError: Skipped 1 previous similar message [125742.157771] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567297692, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4bd3ee9a00/0x8223d616bf875914 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5c9ef031b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [125742.158030] LustreError: 146651:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4fa5618fc0) refcount nonzero (2) after lock cleanup; forcing cleanup. [125742.158032] LustreError: 146651:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [125742.158034] LustreError: 146651:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4fa5618fc0) refcount = 3 [125742.158035] LustreError: 146651:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [125742.158041] LustreError: 146651:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4bd3ee9a00/0x8223d616bf875914 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5c9ef031b expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [125743.568730] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [126006.940501] LustreError: 16611:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [126007.048783] LustreError: 16611:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 827579 previous similar messages [126073.807051] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [126073.916375] Lustre: Skipped 5 previous similar messages [126087.857563] Lustre: oak-MDT0000: haven't heard from client bf016d2e-4371-4096-8391-627cd1e360e7 (at 10.8.30.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8e6920718800, cur 1567298338 expire 1567298188 last 1567298111 [126391.136799] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [126391.301238] LustreError: Skipped 1 previous similar message [126391.368972] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567298341, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d263ec200/0x8223d616c6103668 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ca15ebb2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [126391.369179] LustreError: 147298:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e4e77bfa6c0) refcount nonzero (2) after lock cleanup; forcing cleanup. [126391.369182] LustreError: 147298:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e4e77bfa6c0) refcount = 3 [126391.369183] LustreError: 147298:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [126391.369188] LustreError: 147298:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4d263ec200/0x8223d616c6103668 lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ca15ebb2 expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [126392.652940] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [126420.767974] Lustre: oak-MDT0000: Client dab27dd5-fc23-41f3-deb6-5bdd3dc478d8 (at 10.8.30.17@o2ib6) reconnecting [126688.815577] LustreError: 37925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [126688.923863] LustreError: 37925:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1126260 previous similar messages [126726.971519] LustreError: 147682:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e56e6b1d800) refcount = 1 [126727.129728] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [126727.239045] Lustre: Skipped 3 previous similar messages [127060.367331] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail [127060.531769] LustreError: Skipped 1 previous similar message [127060.599498] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1567299010, 300s ago), entering recovery for MGS@10.0.2.51@o2ib5 ns: MGC10.0.2.51@o2ib5 lock: ffff8e4b6a754200/0x8223d616cb93905d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0xd1fd47d5ca34ea6a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [127060.599715] LustreError: 148157:0:(ldlm_resource.c:1101:ldlm_resource_complain()) MGC10.0.2.51@o2ib5: namespace resource [0x6b616f:0x2:0x0].0x0 (ffff8e50e6882480) refcount nonzero (2) after lock cleanup; forcing cleanup. [127060.599717] LustreError: 148157:0:(ldlm_resource.c:1101:ldlm_resource_complain()) Skipped 1 previous similar message [127060.599719] LustreError: 148157:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e50e6882480) refcount = 3 [127060.599721] LustreError: 148157:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [127060.599725] LustreError: 148157:0:(ldlm_resource.c:1706:ldlm_resource_dump()) ### ### ns: MGC10.0.2.51@o2ib5 lock: ffff8e4b6a754200/0x8223d616cb93905d lrc: 4/1,0 mode: --/CR res: [0x6b616f:0x2:0x0].0x0 rrc: 4 type: PLN flags: 0x1106400000000 nid: local remote: 0xd1fd47d5ca34ea6a expref: -99 pid: 16175 timeout: 0 lvb_type: 0 [127062.010452] LustreError: 16175:0:(ldlm_request.c:148:ldlm_expired_completion_wait()) Skipped 1 previous similar message [127288.854061] LustreError: 37944:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Unknown attr bits: 0x60000 [127288.962345] LustreError: 37944:0:(mdt_lib.c:961:mdt_attr_valid_xlate()) Skipped 1040467 previous similar messages [127371.723500] LustreError: 148609:0:(ldlm_resource.c:1683:ldlm_resource_dump()) --- Resource: [0x6b616f:0x2:0x0].0x0 (ffff8e52ab9406c0) refcount = 2 [127371.881705] LustreError: 148609:0:(ldlm_resource.c:1704:ldlm_resource_dump()) Waiting locks: [127371.983764] Lustre: MGC10.0.2.51@o2ib5: Connection restored to 10.0.2.51@o2ib5 (at 10.0.2.51@o2ib5) [127372.093097] Lustre: Skipped 1 previous similar message [127500.720602] LustreError: 33146:0:(lod_object.c:3919:lod_ah_init()) ASSERTION( !lod_obj_is_striped(child) ) failed: [127500.846575] LustreError: 33146:0:(lod_object.c:3919:lod_ah_init()) LBUG [127500.926853] Pid: 33146, comm: mdt00_037 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019 [127501.049798] Call Trace: [127501.080113] [<ffffffffc0c5a7cc>] libcfs_call_trace+0x8c/0xc0 [libcfs] [127501.159395] [<ffffffffc0c5a87c>] lbug_with_loc+0x4c/0xa0 [libcfs] [127501.235068] [<ffffffffc1675b4f>] lod_ah_init+0x23f/0xde0 [lod] [127501.307355] [<ffffffffc16f209b>] mdd_object_make_hint+0xcb/0x190 [mdd] [127501.387734] [<ffffffffc16ddd50>] mdd_create_data+0x330/0x730 [mdd] [127501.464001] [<ffffffffc15b03fc>] mdt_mfd_open+0xc5c/0xe70 [mdt] [127501.537163] [<ffffffffc15b0b8b>] mdt_finish_open+0x57b/0x690 [mdt] [127501.613324] [<ffffffffc15b309d>] mdt_reint_open+0x23fd/0x3190 [mdt] [127501.690514] [<ffffffffc15a7ca3>] mdt_reint_rec+0x83/0x210 [mdt] [127501.763537] [<ffffffffc158918b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] [127501.842813] [<ffffffffc15896b2>] mdt_intent_reint+0x162/0x430 [mdt] [127501.920001] [<ffffffffc158c4bb>] mdt_intent_opc+0x1eb/0xaf0 [mdt] [127501.995104] [<ffffffffc1594d58>] mdt_intent_policy+0x138/0x320 [mdt] [127502.073330] [<ffffffffc0f782dd>] ldlm_lock_enqueue+0x38d/0x980 [ptlrpc] [127502.154725] [<ffffffffc0fa1c03>] ldlm_handle_enqueue0+0xa83/0x1670 [ptlrpc] [127502.240260] [<ffffffffc1027892>] tgt_enqueue+0x62/0x210 [ptlrpc] [127502.314355] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [127502.397805] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [127502.490622] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [127502.566798] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [127502.626299] [<ffffffff9bd77c37>] ret_from_fork_nospec_end+0x0/0x39 [127502.702443] [<ffffffffffffffff>] 0xffffffffffffffff [127502.763000] Kernel panic - not syncing: LBUG [127502.815109] CPU: 2 PID: 33146 Comm: mdt00_037 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 [127502.967061] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.6.0 10/26/2017 [127503.057649] Call Trace: [127503.087926] [<ffffffff9bd65147>] dump_stack+0x19/0x1b [127503.150440] [<ffffffff9bd5e850>] panic+0xe8/0x21f [127503.208795] [<ffffffffc0c5a8cb>] lbug_with_loc+0x9b/0xa0 [libcfs] [127503.283797] [<ffffffffc1675b4f>] lod_ah_init+0x23f/0xde0 [lod] [127503.355673] [<ffffffffc16f209b>] mdd_object_make_hint+0xcb/0x190 [mdd] [127503.435865] [<ffffffffc16ddd50>] mdd_create_data+0x330/0x730 [mdd] [127503.511907] [<ffffffffc15b03fc>] mdt_mfd_open+0xc5c/0xe70 [mdt] [127503.584819] [<ffffffffc15b0b8b>] mdt_finish_open+0x57b/0x690 [mdt] [127503.660851] [<ffffffffc15b309d>] mdt_reint_open+0x23fd/0x3190 [mdt] [127503.737931] [<ffffffffc0ce55cc>] ? cfs_match_nid+0x9c/0xd0 [lnet] [127503.812923] [<ffffffffc1597d13>] ? ucred_set_jobid+0x53/0x70 [mdt] [127503.888956] [<ffffffffc15a7ca3>] mdt_reint_rec+0x83/0x210 [mdt] [127503.961868] [<ffffffffc158918b>] mdt_reint_internal+0x5fb/0x9c0 [mdt] [127504.041020] [<ffffffffc15896b2>] mdt_intent_reint+0x162/0x430 [mdt] [127504.118092] [<ffffffffc158c4bb>] mdt_intent_opc+0x1eb/0xaf0 [mdt] [127504.193120] [<ffffffffc0fc9f90>] ? lustre_swab_ldlm_policy_data+0x30/0x30 [ptlrpc] [127504.285794] [<ffffffffc1594d58>] mdt_intent_policy+0x138/0x320 [mdt] [127504.363921] [<ffffffffc0f782dd>] ldlm_lock_enqueue+0x38d/0x980 [ptlrpc] [127504.445171] [<ffffffffc0fa1c03>] ldlm_handle_enqueue0+0xa83/0x1670 [ptlrpc] [127504.530582] [<ffffffffc0fca010>] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] [127504.621199] [<ffffffffc1027892>] tgt_enqueue+0x62/0x210 [ptlrpc] [127504.695177] [<ffffffffc102b7ca>] tgt_request_handle+0x92a/0x1370 [ptlrpc] [127504.778509] [<ffffffffc0fd405b>] ptlrpc_server_handle_request+0x23b/0xaa0 [ptlrpc] [127504.871207] [<ffffffffc0fd0688>] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [127504.953481] [<ffffffff9b6d7c52>] ? default_wake_function+0x12/0x20 [127505.029512] [<ffffffff9b6ccceb>] ? __wake_up_common+0x5b/0x90 [127505.100378] [<ffffffffc0fd77a2>] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [127505.176442] [<ffffffffc0fd6d10>] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [127505.265996] [<ffffffff9b6c2e81>] kthread+0xd1/0xe0 [127505.325387] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40 [127505.399341] [<ffffffff9bd77c37>] ret_from_fork_nospec_begin+0x21/0x21 [127505.478492] [<ffffffff9b6c2db0>] ? insert_kthread_work+0x40/0x40