<6>[ 0.000000] Initializing cgroup subsys cpuset <6>[ 0.000000] Initializing cgroup subsys cpu <5>[ 0.000000] Linux version 2.6.32-573.12.1.el6.atlas.x86_64 (jsimmons@lgmgmt1) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Mon Jan 25 12:40:27 EST 2016 <6>[ 0.000000] Command line: initrd=initrd-2.6.32-573.12.1.el6.atlas.x86_64-gedi selinux=0 audit=0 panic=10 console=tty0 console=ttyS1,115200n8 console=ttyS2,115200n8 crashkernel=192M init=/gedi-preinit BOOT_IMAGE=vmlinuz-2.6.32-573.12.1.el6.atlas.x86_64 BOOTIF=01-20-47-47-7e-c5-e0 <6>[ 0.000000] KERNEL supported cpus: <6>[ 0.000000] Intel GenuineIntel <6>[ 0.000000] AMD AuthenticAMD <6>[ 0.000000] Centaur CentaurHauls <6>[ 0.000000] BIOS-provided physical RAM map: <6>[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009d000 (usable) <6>[ 0.000000] BIOS-e820: 000000000009d000 - 00000000000a0000 (reserved) <6>[ 0.000000] BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) <6>[ 0.000000] BIOS-e820: 0000000000100000 - 0000000079f89000 (usable) <6>[ 0.000000] BIOS-e820: 0000000079f89000 - 000000007ab0b000 (reserved) <6>[ 0.000000] BIOS-e820: 000000007ab0b000 - 000000007b53b000 (ACPI NVS) <6>[ 0.000000] BIOS-e820: 000000007b53b000 - 000000007b6bf000 (ACPI data) <6>[ 0.000000] BIOS-e820: 000000007b6bf000 - 000000007b700000 (usable) <6>[ 0.000000] BIOS-e820: 000000007b700000 - 0000000090000000 (reserved) <6>[ 0.000000] BIOS-e820: 00000000feda8000 - 00000000fedac000 (reserved) <6>[ 0.000000] BIOS-e820: 00000000ff310000 - 0000000100000000 (reserved) <6>[ 0.000000] BIOS-e820: 0000000100000000 - 0000004080000000 (usable) <6>[ 0.000000] DMI 2.8 present. <4>[ 0.000000] SMBIOS version 2.8 @ 0xFE300 <7>[ 0.000000] DMI: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.3.6 06/03/2015 <7>[ 0.000000] e820 update range: 0000000000000000 - 0000000000001000 (usable) ==> (reserved) <7>[ 0.000000] e820 remove range: 00000000000a0000 - 0000000000100000 (usable) <6>[ 0.000000] No AGP bridge found <6>[ 0.000000] last_pfn = 0x4080000 max_arch_pfn = 0x400000000 <6>[ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 <6>[ 0.000000] last_pfn = 0x7b700 max_arch_pfn = 0x400000000 <7>[ 0.000000] initial memory mapped : 0 - 20000000 <6>[ 0.000000] Using GB pages for direct mapping <6>[ 0.000000] init_memory_mapping: 0000000000000000-000000007b700000 <7>[ 0.000000] 0000000000 - 0040000000 page 1G <7>[ 0.000000] 0040000000 - 007b600000 page 2M <7>[ 0.000000] 007b600000 - 007b700000 page 4k <7>[ 0.000000] kernel direct mapping tables up to 7b700000 @ 8000-b000 <7>[ 0.000000] Use unified mapping for non-reserved e820 regions. <6>[ 0.000000] init_memory_mapping: 0000000100000000-0000004080000000 <7>[ 0.000000] 0100000000 - 4080000000 page 1G <7>[ 0.000000] kernel direct mapping tables up to 4080000000 @ a000-b000 <6>[ 0.000000] RAMDISK: 79064000 - 79f681a0 <6>[ 0.000000] ACPI: Deleted _OSI() <6>[ 0.000000] ACPI: Deleted _OSI() <4>[ 0.000000] ACPI: RSDP 00000000000fe320 00024 (v02 DELL ) <4>[ 0.000000] ACPI: XSDT 000000007b6bd0e8 000C4 (v01 DELL PE_SC3 00000000 01000013) <4>[ 0.000000] ACPI: FACP 000000007b6b9000 000F4 (v04 DELL PE_SC3 00000000 DELL 00000001) <4>[ 0.000000] ACPI: DSDT 000000007b6a2000 0F103 (v02 DELL PE_SC3 00000003 DELL 00000001) <4>[ 0.000000] ACPI: FACS 000000007b538000 00040 <4>[ 0.000000] ACPI: MCEJ 000000007b6bc000 00130 (v01 INTEL 00000001 INTL 0100000D) <4>[ 0.000000] ACPI: WD__ 000000007b6bb000 00134 (v01 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: SLIC 000000007b6ba000 00024 (v01 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: HPET 000000007b6b8000 00038 (v01 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: APIC 000000007b6b7000 0085C (v02 DELL PE_SC3 00000000 DELL 00000001) <4>[ 0.000000] ACPI: MCFG 000000007b6b6000 0003C (v01 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: MSCT 000000007b6b5000 00090 (v01 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: PMCT 000000007b6b4000 00064 (v01 DELL PE_SC3 00000000 DELL 00000001) <4>[ 0.000000] ACPI: SLIT 000000007b6b3000 0006C (v01 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: SRAT 000000007b6b2000 00E30 (v03 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: SSDT 000000007b5d6000 CBC9B (v02 DELL PE_SC3 00004000 INTL 20121114) <4>[ 0.000000] ACPI: SSDT 000000007b5d4000 01C94 (v02 DELL PE_SC3 00000002 INTL 20121114) <4>[ 0.000000] ACPI: SSDT 000000007b5d3000 0006E (v02 DELL PE_SC3 00000002 INTL 20121114) <4>[ 0.000000] ACPI: PRAD 000000007b5d2000 00132 (v02 DELL PE_SC3 00000002 INTL 20121114) <4>[ 0.000000] ACPI: DMAR 000000007b5d1000 000B0 (v01 DELL PE_SC3 00000001 DELL 00000001) <4>[ 0.000000] ACPI: HEST 000000007b5d0000 0017C (v01 DELL PE_SC3 00000002 DELL 00000001) <4>[ 0.000000] ACPI: BERT 000000007b5cf000 00030 (v01 DELL PE_SC3 00000002 DELL 00000001) <4>[ 0.000000] ACPI: ERST 000000007b5ce000 00230 (v01 DELL PE_SC3 00000002 DELL 00000001) <4>[ 0.000000] ACPI: EINJ 000000007b5cd000 00150 (v01 DELL PE_SC3 00000002 DELL 00000001) <7>[ 0.000000] ACPI: Local APIC address 0xfee00000 <6>[ 0.000000] Setting APIC routing to flat. <6>[ 0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0 <6>[ 0.000000] SRAT: PXM 0 -> APIC 2 -> Node 0 <6>[ 0.000000] SRAT: PXM 0 -> APIC 4 -> Node 0 <6>[ 0.000000] SRAT: PXM 0 -> APIC 6 -> Node 0 <6>[ 0.000000] SRAT: PXM 0 -> APIC 8 -> Node 0 <6>[ 0.000000] SRAT: PXM 0 -> APIC 10 -> Node 0 <6>[ 0.000000] SRAT: PXM 0 -> APIC 12 -> Node 0 <6>[ 0.000000] SRAT: PXM 0 -> APIC 14 -> Node 0 <6>[ 0.000000] SRAT: Node 0 PXM 0 0-4080000000 <7>[ 0.000000] NUMA: Using 63 for the hash shift. <6>[ 0.000000] Bootmem setup node 0 0000000000000000-0000004080000000 <6>[ 0.000000] NODE_DATA [000000000000a000 - 000000000003dfff] <6>[ 0.000000] bootmap [0000000000100000 - 000000000090ffff] pages 810 <6>[ 0.000000] (8 early reservations) ==> bootmem [0000000000 - 4080000000] <6>[ 0.000000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] <6>[ 0.000000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] <6>[ 0.000000] #2 [0001000000 - 000203db64] TEXT DATA BSS ==> [0001000000 - 000203db64] <6>[ 0.000000] #3 [0079064000 - 0079f681a0] RAMDISK ==> [0079064000 - 0079f681a0] <6>[ 0.000000] #4 [000009d800 - 0000100000] BIOS reserved ==> [000009d800 - 0000100000] <6>[ 0.000000] #5 [000203e000 - 000203e98d] BRK ==> [000203e000 - 000203e98d] <6>[ 0.000000] #6 [0000008000 - 000000a000] PGTABLE ==> [0000008000 - 000000a000] <6>[ 0.000000] #7 [0000001000 - 000000106c] ACPI SLIT ==> [0000001000 - 000000106c] <6>[ 0.000000] Reserving 192MB of memory at 48MB for crashkernel (System RAM: 264192MB) <7>[ 0.000000] [ffffea0000000000-ffffea003fffffff] PMD -> [ffff880028600000-ffff8800669fffff] on node 0 <7>[ 0.000000] [ffffea0040000000-ffffea00523fffff] PMD -> [ffff880066c00000-ffff880078ffffff] on node 0 <7>[ 0.000000] [ffffea0052400000-ffffea007fffffff] PMD -> [ffff880100000000-ffff88012dbfffff] on node 0 <7>[ 0.000000] [ffffea0080000000-ffffea00bfffffff] PMD -> [ffff88012de00000-ffff88016ddfffff] on node 0 <7>[ 0.000000] [ffffea00c0000000-ffffea00e1bfffff] PMD -> [ffff88016e000000-ffff88018fbfffff] on node 0 <4>[ 0.000000] Zone PFN ranges: <4>[ 0.000000] DMA 0x00000001 -> 0x00001000 <4>[ 0.000000] DMA32 0x00001000 -> 0x00100000 <4>[ 0.000000] Normal 0x00100000 -> 0x04080000 <4>[ 0.000000] Movable zone start PFN for each node <4>[ 0.000000] early_node_map[4] active PFN ranges <4>[ 0.000000] 0: 0x00000001 -> 0x0000009d <4>[ 0.000000] 0: 0x00000100 -> 0x00079f89 <4>[ 0.000000] 0: 0x0007b6bf -> 0x0007b700 <4>[ 0.000000] 0: 0x00100000 -> 0x04080000 <7>[ 0.000000] On node 0 totalpages: 67084134 <7>[ 0.000000] DMA zone: 56 pages used for memmap <7>[ 0.000000] DMA zone: 103 pages reserved <7>[ 0.000000] DMA zone: 3837 pages, LIFO batch:0 <7>[ 0.000000] DMA32 zone: 14280 pages used for memmap <7>[ 0.000000] DMA32 zone: 481282 pages, LIFO batch:31 <7>[ 0.000000] Normal zone: 910336 pages used for memmap <7>[ 0.000000] Normal zone: 65674240 pages, LIFO batch:31 <6>[ 0.000000] ACPI: PM-Timer IO Port: 0x408 <7>[ 0.000000] ACPI: Local APIC address 0xfee00000 <6>[ 0.000000] Setting APIC routing to flat. <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0c] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC (acpi_id[0xff] lapic_id[0xff] disabled) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x04] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x05] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x06] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x07] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x08] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x09] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x0f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x14] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x15] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x16] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x17] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x18] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x19] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x1f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x20] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x21] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x22] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x23] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x24] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x25] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x26] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x27] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x28] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x29] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x2f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x30] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x31] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x32] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x33] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x34] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x35] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x36] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x37] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x38] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x39] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x3f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x40] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x41] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x42] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x43] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x44] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x45] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x46] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x47] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x48] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x49] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x4f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x50] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x51] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x52] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x53] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x54] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x55] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x56] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x57] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x58] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x59] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x5f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x60] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x61] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x62] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x63] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x64] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x65] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x66] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x67] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x68] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x69] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x6f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x70] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x71] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x72] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x73] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x74] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x75] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x76] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x77] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x78] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x79] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x7f] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x80] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x81] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x82] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x83] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x84] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x85] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x86] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x87] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x88] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x89] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8a] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8b] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8c] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8d] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8e] high level lint[0x1]) <6>[ 0.000000] ACPI: LAPIC_NMI (acpi_id[0x8f] high level lint[0x1]) <6>[ 0.000000] ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) <6>[ 0.000000] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 <6>[ 0.000000] ACPI: IOAPIC (id[0x09] address[0xfec01000] gsi_base[24]) <6>[ 0.000000] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-47 <6>[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) <6>[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) <7>[ 0.000000] ACPI: IRQ0 used by override. <7>[ 0.000000] ACPI: IRQ2 used by override. <7>[ 0.000000] ACPI: IRQ9 used by override. <6>[ 0.000000] Using ACPI (MADT) for SMP configuration information <6>[ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 <6>[ 0.000000] SMP: Allowing 144 CPUs, 136 hotplug CPUs <7>[ 0.000000] nr_irqs_gsi: 48 <6>[ 0.000000] PM: Registered nosave memory: 000000000009d000 - 00000000000a0000 <6>[ 0.000000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000 <6>[ 0.000000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000 <6>[ 0.000000] PM: Registered nosave memory: 0000000079f89000 - 000000007ab0b000 <6>[ 0.000000] PM: Registered nosave memory: 000000007ab0b000 - 000000007b53b000 <6>[ 0.000000] PM: Registered nosave memory: 000000007b53b000 - 000000007b6bf000 <6>[ 0.000000] PM: Registered nosave memory: 000000007b700000 - 0000000090000000 <6>[ 0.000000] PM: Registered nosave memory: 0000000090000000 - 00000000feda8000 <6>[ 0.000000] PM: Registered nosave memory: 00000000feda8000 - 00000000fedac000 <6>[ 0.000000] PM: Registered nosave memory: 00000000fedac000 - 00000000ff310000 <6>[ 0.000000] PM: Registered nosave memory: 00000000ff310000 - 0000000100000000 <6>[ 0.000000] Allocating PCI resources starting at 90000000 (gap: 90000000:6eda8000) <6>[ 0.000000] Booting paravirtualized kernel on bare hardware <6>[ 0.000000] NR_CPUS:4096 nr_cpumask_bits:144 nr_cpu_ids:144 nr_node_ids:1 <6>[ 0.000000] PERCPU: Embedded 30 pages/cpu @ffff88018fc00000 s91288 r8192 d23400 u131072 <6>[ 0.000000] pcpu-alloc: s91288 r8192 d23400 u131072 alloc=1*2097152 <6>[ 0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 <6>[ 0.000000] pcpu-alloc: [0] 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 <6>[ 0.000000] pcpu-alloc: [0] 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 <6>[ 0.000000] pcpu-alloc: [0] 048 049 050 051 052 053 054 055 056 057 058 059 060 061 062 063 <6>[ 0.000000] pcpu-alloc: [0] 064 065 066 067 068 069 070 071 072 073 074 075 076 077 078 079 <6>[ 0.000000] pcpu-alloc: [0] 080 081 082 083 084 085 086 087 088 089 090 091 092 093 094 095 <6>[ 0.000000] pcpu-alloc: [0] 096 097 098 099 100 101 102 103 104 105 106 107 108 109 110 111 <6>[ 0.000000] pcpu-alloc: [0] 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 <6>[ 0.000000] pcpu-alloc: [0] 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 <4>[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 66159359 <4>[ 0.000000] Policy zone: Normal <5>[ 0.000000] Kernel command line: initrd=initrd-2.6.32-573.12.1.el6.atlas.x86_64-gedi selinux=0 audit=0 panic=10 console=tty0 console=ttyS1,115200n8 console=ttyS2,115200n8 crashkernel=192M init=/gedi-preinit BOOT_IMAGE=vmlinuz-2.6.32-573.12.1.el6.atlas.x86_64 BOOTIF=01-20-47-47-7e-c5-e0 <6>[ 0.000000] audit: disabled (until reboot) <6>[ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) <6>[ 0.000000] xsave/xrstor: enabled xstate_bv 0x7, cntxt size 0x340 <6>[ 0.000000] Checking aperture... <6>[ 0.000000] No AGP bridge found <6>[ 0.000000] dmar: Queued invalidation will be enabled to support x2apic and Intr-remapping. <6>[ 0.000000] Memory: 264353860k/270532608k available (5398k kernel code, 2196072k absent, 3982676k reserved, 7010k data, 1300k init) <6>[ 0.000000] Hierarchical RCU implementation. <6>[ 0.000000] NR_IRQS:33024 nr_irqs:1968 <6>[ 0.000000] Extended CMOS year: 2000 <4>[ 0.000000] Console: colour VGA+ 80x25 <6>[ 0.000000] console [tty0] enabled <6>[ 0.000000] console [ttyS1] enabled <6>[ 0.000000] allocated 1073741824 bytes of page_cgroup <6>[ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups <7>[ 0.000000] hpet clockevent registered <4>[ 0.000000] Fast TSC calibration using PIT <4>[ 0.001000] Detected 3199.802 MHz processor. <6>[ 0.000010] Calibrating delay loop (skipped), value calculated using timer frequency.. 6399.60 BogoMIPS (lpj=3199802) <6>[ 0.012469] pid_max: default: 147456 minimum: 1152 <6>[ 0.018243] Security Framework initialized <6>[ 0.023121] SELinux: Disabled at boot. <6>[ 0.044085] Dentry cache hash table entries: 33554432 (order: 16, 268435456 bytes) <6>[ 0.085886] Inode-cache hash table entries: 16777216 (order: 15, 134217728 bytes) <4>[ 0.107183] Mount-cache hash table entries: 256 <6>[ 0.112678] Initializing cgroup subsys ns <6>[ 0.117453] Initializing cgroup subsys cpuacct <6>[ 0.122724] Initializing cgroup subsys memory <6>[ 0.127897] Initializing cgroup subsys devices <6>[ 0.133154] Initializing cgroup subsys freezer <6>[ 0.138411] Initializing cgroup subsys net_cls <6>[ 0.143659] Initializing cgroup subsys blkio <6>[ 0.148726] Initializing cgroup subsys perf_event <6>[ 0.154280] Initializing cgroup subsys net_prio <6>[ 0.159672] CPU: Physical Processor ID: 0 <6>[ 0.164442] CPU: Processor Core ID: 0 <6>[ 0.169551] mce: CPU supports 22 MCE banks <6>[ 0.174446] CPU0: Thermal monitoring enabled (TM1) <6>[ 0.180114] using mwait in idle threads. <6>[ 0.185503] ACPI: Core revision 20090903 <6>[ 0.233255] ftrace: converting mcount calls to 0f 1f 44 00 00 <6>[ 0.239974] ftrace: allocating 22138 entries in 87 pages <6>[ 0.250876] dmar: Host address width 46 <6>[ 0.255456] dmar: DRHD base: 0x000000fbffc000 flags: 0x1 <6>[ 0.261694] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020df <6>[ 0.271248] dmar: RMRR base: 0x0000007b800000 end: 0x0000007b8fffff <6>[ 0.278549] dmar: ATSR flags: 0x0 <6>[ 0.282634] IOAPIC id 8 under DRHD base 0xfbffc000 <6>[ 0.288276] IOAPIC id 9 under DRHD base 0xfbffc000 <6>[ 0.293922] HPET id 0 under DRHD base 0xfbffc000 <4>[ 0.299368] Your BIOS is broken and requested that x2apic be disabled. <4>[ 0.299368] This will slightly decrease performance. <4>[ 0.299369] Use 'intremap=no_x2apic_optout' to override BIOS request. <6>[ 0.320350] Enabled IRQ remapping in xapic mode <6>[ 0.325711] x2apic not enabled, IRQ remapping is in xapic mode <6>[ 0.332526] APIC routing finalized to physical flat. <7>[ 0.338661] alloc irq_desc for 48 on node 0 <7>[ 0.338662] alloc kstat_irqs on node 0 <7>[ 0.338670] alloc irq_2_iommu on node 0 <7>[ 0.338673] alloc irq_2_iommu on node 0 <7>[ 0.338676] alloc irq_2_iommu on node 0 <7>[ 0.338679] alloc irq_2_iommu on node 0 <7>[ 0.338682] alloc irq_2_iommu on node 0 <7>[ 0.338685] alloc irq_2_iommu on node 0 <7>[ 0.338688] alloc irq_2_iommu on node 0 <7>[ 0.338690] alloc irq_2_iommu on node 0 <7>[ 0.338693] alloc irq_2_iommu on node 0 <7>[ 0.338696] alloc irq_2_iommu on node 0 <7>[ 0.338699] alloc irq_2_iommu on node 0 <7>[ 0.338702] alloc irq_2_iommu on node 0 <7>[ 0.338705] alloc irq_2_iommu on node 0 <7>[ 0.338708] alloc irq_2_iommu on node 0 <7>[ 0.338711] alloc irq_2_iommu on node 0 <6>[ 0.338852] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 <6>[ 0.355856] CPU0: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz stepping 02 <6>[ 0.464949] Performance Events: PEBS fmt2+, 16-deep LBR, Haswell events, full-width counters, Intel PMU driver. <6>[ 0.477144] ... version: 3 <6>[ 0.481913] ... bit width: 48 <6>[ 0.486777] ... generic registers: 8 <6>[ 0.491549] ... value mask: 0000ffffffffffff <6>[ 0.497777] ... max period: 0000ffffffffffff <6>[ 0.504006] ... fixed-purpose events: 3 <6>[ 0.508765] ... event mask: 00000007000000ff <6>[ 0.515917] NMI watchdog enabled, takes one hw-pmu counter. <6>[ 0.522499] Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 <6>[ 0.753579] Brought up 8 CPUs <6>[ 0.757675] Total of 8 processors activated (51196.83 BogoMIPS). <7>[ 0.770406] sizeof(vma)=200 bytes <7>[ 0.770408] sizeof(page)=56 bytes <7>[ 0.770408] sizeof(inode)=592 bytes <7>[ 0.770409] sizeof(dentry)=192 bytes <7>[ 0.770409] sizeof(ext3inode)=800 bytes <7>[ 0.770410] sizeof(buffer_head)=104 bytes <7>[ 0.770410] sizeof(skbuff)=232 bytes <7>[ 0.770411] sizeof(task_struct)=2672 bytes <6>[ 0.770928] devtmpfs: initialized <6>[ 0.782684] PM: Registering ACPI NVS region at 7ab0b000 (10682368 bytes) <6>[ 0.790807] regulator: core version 0.5 <6>[ 0.795453] NET: Registered protocol family 16 <6>[ 0.801325] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it <6>[ 0.810302] ACPI: bus type pci registered <5>[ 0.815145] PCI: MCFG configuration 0: base 80000000 segment 0 buses 0 - 255 <5>[ 0.823323] PCI: MCFG area at 80000000 reserved in E820 <6>[ 0.853202] PCI: Using MMCONFIG at 80000000 - 8fffffff <6>[ 0.859233] PCI: Using configuration type 1 for base access <6>[ 0.865904] PCI: Dell System detected, enabling pci=bfsort. <4>[ 0.875063] bio: create slab at 0 <7>[ 0.882704] ACPI: EC: Look up EC in DSDT <5>[ 1.023071] ACPI: BIOS _OSI(Linux) query ignored <6>[ 1.028852] ACPI: Interpreter enabled <6>[ 1.033242] ACPI: (supports S0 S5) <5>[ 1.037501] ACPI: BIOS offers _GTS <5>[ 1.041590] ACPI: If "acpi.gts=1" improves suspend, please notify linux-acpi@vger.kernel.org <6>[ 1.051536] ACPI: Using IOAPIC for interrupt routing <6>[ 1.061227] ACPI: No dock devices found. <6>[ 1.065922] HEST: Table parsing has been initialized. <6>[ 1.071852] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug <6>[ 1.082680] ACPI: PCI Root Bridge [UNC0] (domain 0000 [bus ff]) <6>[ 1.089607] PCI host bridge to bus 0000:ff <7>[ 1.095672] ACPI: PCI Interrupt Routing Table [\_SB_.UNC0._PRT] <6>[ 1.096030] pci0000:ff: Requesting ACPI _OSC control (0x15) <6>[ 1.102802] pci0000:ff: ACPI _OSC control (0x15) granted <6>[ 2.109715] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-fe]) <6>[ 2.117294] pci_root PNP0A08:00: host bridge window [io 0x0000-0x03bb] <6>[ 2.124976] pci_root PNP0A08:00: host bridge window [io 0x03bc-0x03df] <6>[ 2.132655] pci_root PNP0A08:00: host bridge window [io 0x03e0-0x0cf7] <6>[ 2.140338] pci_root PNP0A08:00: host bridge window [io 0x1000-0xffff] <6>[ 2.148021] pci_root PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff] <6>[ 2.156703] pci_root PNP0A08:00: host bridge window [mem 0x000c4000-0x000cbfff] <6>[ 2.165381] pci_root PNP0A08:00: host bridge window [mem 0x90000000-0xfbffbfff] <6>[ 2.174059] pci_root PNP0A08:00: host bridge window [mem 0x30000000000-0x33fffffffff] <6>[ 2.183305] pci_root PNP0A08:00: ignoring host bridge window [mem 0x000c4000-0x000cbfff] (conflicts with Video ROM [mem 0x000c0000-0x000c7fff]) <6>[ 2.198223] PCI host bridge to bus 0000:00 <6>[ 2.203082] pci_bus 0000:00: root bus resource [io 0x0000-0x03bb] <6>[ 2.210278] pci_bus 0000:00: root bus resource [io 0x03bc-0x03df] <6>[ 2.217478] pci_bus 0000:00: root bus resource [io 0x03e0-0x0cf7] <6>[ 2.224663] pci_bus 0000:00: root bus resource [io 0x1000-0xffff] <6>[ 2.231857] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff] <6>[ 2.239827] pci_bus 0000:00: root bus resource [mem 0x90000000-0xfbffbfff] <6>[ 2.247805] pci_bus 0000:00: root bus resource [mem 0x30000000000-0x33fffffffff] <6>[ 2.256636] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold <6>[ 2.263739] pci 0000:00:01.0: PME# disabled <6>[ 2.268733] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold <6>[ 2.275833] pci 0000:00:02.0: PME# disabled <6>[ 2.280829] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold <6>[ 2.287931] pci 0000:00:03.0: PME# disabled <6>[ 2.292946] pci 0000:00:03.1: PME# supported from D0 D3hot D3cold <6>[ 2.300047] pci 0000:00:03.1: PME# disabled <6>[ 2.310293] pci 0000:00:03.2: PME# supported from D0 D3hot D3cold <6>[ 2.317387] pci 0000:00:03.2: PME# disabled <7>[ 2.322458] pci 0000:00:05.4: reg 10: [mem 0x92604000-0x92604fff] <7>[ 2.322549] pci 0000:00:11.0: reg 30: [mem 0xffff0000-0xffffffff pref] <7>[ 2.322602] pci 0000:00:11.4: reg 10: [io 0x3078-0x307f] <7>[ 2.322609] pci 0000:00:11.4: reg 14: [io 0x308c-0x308f] <7>[ 2.322615] pci 0000:00:11.4: reg 18: [io 0x3070-0x3077] <7>[ 2.322622] pci 0000:00:11.4: reg 1c: [io 0x3088-0x308b] <7>[ 2.322629] pci 0000:00:11.4: reg 20: [io 0x3040-0x305f] <7>[ 2.322636] pci 0000:00:11.4: reg 24: [mem 0x92601000-0x926017ff] <6>[ 2.322665] pci 0000:00:11.4: PME# supported from D3hot <6>[ 2.328794] pci 0000:00:11.4: PME# disabled <7>[ 2.333789] pci 0000:00:16.0: reg 10: [mem 0x33ffff03000-0x33ffff0300f 64bit] <6>[ 2.333833] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold <6>[ 2.340934] pci 0000:00:16.0: PME# disabled <7>[ 2.345926] pci 0000:00:16.1: reg 10: [mem 0x33ffff02000-0x33ffff0200f 64bit] <6>[ 2.345970] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold <6>[ 2.353070] pci 0000:00:16.1: PME# disabled <7>[ 2.358078] pci 0000:00:1a.0: reg 10: [mem 0x92603000-0x926033ff] <6>[ 2.358142] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold <6>[ 2.365247] pci 0000:00:1a.0: PME# disabled <6>[ 2.370263] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold <6>[ 2.377364] pci 0000:00:1c.0: PME# disabled <6>[ 2.382380] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold <6>[ 2.389481] pci 0000:00:1c.7: PME# disabled <7>[ 2.394483] pci 0000:00:1d.0: reg 10: [mem 0x92602000-0x926023ff] <6>[ 2.394547] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold <6>[ 2.401649] pci 0000:00:1d.0: PME# disabled <7>[ 2.406828] pci 0000:00:1f.2: reg 10: [io 0x3068-0x306f] <7>[ 2.406834] pci 0000:00:1f.2: reg 14: [io 0x3084-0x3087] <7>[ 2.406840] pci 0000:00:1f.2: reg 18: [io 0x3060-0x3067] <7>[ 2.406845] pci 0000:00:1f.2: reg 1c: [io 0x3080-0x3083] <7>[ 2.406851] pci 0000:00:1f.2: reg 20: [io 0x3020-0x303f] <7>[ 2.406857] pci 0000:00:1f.2: reg 24: [mem 0x92600000-0x926007ff] <6>[ 2.406880] pci 0000:00:1f.2: PME# supported from D3hot <6>[ 2.413010] pci 0000:00:1f.2: PME# disabled <6>[ 2.417993] pci 0000:00:01.0: PCI bridge to [bus 03-03] <7>[ 2.424618] pci 0000:04:00.0: reg 10: [mem 0x92500000-0x925fffff 64bit] <7>[ 2.424838] pci 0000:04:00.0: reg 18: [mem 0x91000000-0x917fffff 64bit pref] <7>[ 2.425245] pci 0000:04:00.0: reg 30: [mem 0xfff00000-0xffffffff pref] <6>[ 2.426171] pci 0000:00:02.0: PCI bridge to [bus 04-04] <7>[ 2.432300] pci 0000:00:02.0: bridge window [mem 0x92500000-0x925fffff] <7>[ 2.432303] pci 0000:00:02.0: bridge window [mem 0x91000000-0x917fffff 64bit pref] <7>[ 2.432330] pci 0000:02:00.0: reg 10: [mem 0x92230000-0x9223ffff 64bit pref] <7>[ 2.432339] pci 0000:02:00.0: reg 18: [mem 0x92240000-0x9224ffff 64bit pref] <7>[ 2.432349] pci 0000:02:00.0: reg 20: [mem 0x92250000-0x9225ffff 64bit pref] <7>[ 2.432355] pci 0000:02:00.0: reg 30: [mem 0xfffc0000-0xffffffff pref] <6>[ 2.432381] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold <6>[ 2.439481] pci 0000:02:00.0: PME# disabled <7>[ 2.444476] pci 0000:02:00.1: reg 10: [mem 0x92200000-0x9220ffff 64bit pref] <7>[ 2.444486] pci 0000:02:00.1: reg 18: [mem 0x92210000-0x9221ffff 64bit pref] <7>[ 2.444495] pci 0000:02:00.1: reg 20: [mem 0x92220000-0x9222ffff 64bit pref] <7>[ 2.444502] pci 0000:02:00.1: reg 30: [mem 0xfffc0000-0xffffffff pref] <6>[ 2.444527] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold <6>[ 2.451650] pci 0000:02:00.1: PME# disabled <6>[ 2.456640] pci 0000:00:03.0: PCI bridge to [bus 02-02] <7>[ 2.462771] pci 0000:00:03.0: bridge window [mem 0x92200000-0x922fffff 64bit pref] <7>[ 2.462799] pci 0000:01:00.0: reg 10: [mem 0x92330000-0x9233ffff 64bit pref] <7>[ 2.462808] pci 0000:01:00.0: reg 18: [mem 0x92340000-0x9234ffff 64bit pref] <7>[ 2.462818] pci 0000:01:00.0: reg 20: [mem 0x92350000-0x9235ffff 64bit pref] <7>[ 2.462825] pci 0000:01:00.0: reg 30: [mem 0xfffc0000-0xffffffff pref] <6>[ 2.462850] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold <6>[ 2.469951] pci 0000:01:00.0: PME# disabled <7>[ 2.474948] pci 0000:01:00.1: reg 10: [mem 0x92300000-0x9230ffff 64bit pref] <7>[ 2.474958] pci 0000:01:00.1: reg 18: [mem 0x92310000-0x9231ffff 64bit pref] <7>[ 2.474967] pci 0000:01:00.1: reg 20: [mem 0x92320000-0x9232ffff 64bit pref] <7>[ 2.474974] pci 0000:01:00.1: reg 30: [mem 0xfffc0000-0xffffffff pref] <6>[ 2.474999] pci 0000:01:00.1: PME# supported from D0 D3hot D3cold <6>[ 2.482103] pci 0000:01:00.1: PME# disabled <6>[ 2.487097] pci 0000:00:03.1: PCI bridge to [bus 01-01] <7>[ 2.493319] pci 0000:00:03.1: bridge window [mem 0x92300000-0x923fffff 64bit pref] <7>[ 2.493335] pci 0000:05:00.0: reg 10: [io 0x2000-0x20ff] <7>[ 2.493341] pci 0000:05:00.0: reg 14: [mem 0x92400000-0x9240ffff 64bit] <7>[ 2.493354] pci 0000:05:00.0: reg 30: [mem 0xfff00000-0xffffffff pref] <7>[ 2.493370] pci 0000:05:00.0: supports D1 D2 <7>[ 2.493384] pci 0000:05:00.0: reg 174: [mem 0x00000000-0x0000ffff 64bit] <6>[ 2.493414] pci 0000:00:03.2: PCI bridge to [bus 05-05] <7>[ 2.499542] pci 0000:00:03.2: bridge window [io 0x2000-0x2fff] <7>[ 2.499544] pci 0000:00:03.2: bridge window [mem 0x92400000-0x924fffff] <6>[ 2.499573] pci 0000:00:1c.0: PCI bridge to [bus 06-06] <6>[ 2.507732] pci 0000:07:00.0: PME# supported from D0 D3hot D3cold <6>[ 2.514909] pci 0000:07:00.0: PME# disabled <6>[ 2.520408] pci 0000:00:1c.7: PCI bridge to [bus 07-0b] <7>[ 2.526539] pci 0000:00:1c.7: bridge window [mem 0x91800000-0x921fffff] <7>[ 2.526543] pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <6>[ 2.529068] pci 0000:08:00.0: PME# supported from D0 D3hot D3cold <6>[ 2.536250] pci 0000:08:00.0: PME# disabled <6>[ 2.543536] pci 0000:07:00.0: PCI bridge to [bus 08-0b] <7>[ 2.549868] pci 0000:07:00.0: bridge window [mem 0x91800000-0x921fffff] <7>[ 2.550000] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <6>[ 2.555054] pci 0000:08:00.0: PCI bridge to [bus 09-0a] <7>[ 2.561425] pci 0000:08:00.0: bridge window [mem 0x91800000-0x920fffff] <7>[ 2.561581] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <7>[ 2.562475] pci 0000:0a:00.0: reg 10: [mem 0x90000000-0x90ffffff pref] <7>[ 2.562756] pci 0000:0a:00.0: reg 14: [mem 0x92000000-0x92003fff] <7>[ 2.563036] pci 0000:0a:00.0: reg 18: [mem 0x91800000-0x91ffffff] <6>[ 2.567095] pci 0000:09:00.0: PCI bridge to [bus 0a-0a] <7>[ 2.573463] pci 0000:09:00.0: bridge window [mem 0x91800000-0x920fffff] <7>[ 2.573621] pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <7>[ 2.574452] pci_bus 0000:00: on NUMA node 0 <7>[ 2.574454] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] <7>[ 2.576736] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR1A._PRT] <7>[ 2.576763] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR2A._PRT] <7>[ 2.576791] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR3A._PRT] <7>[ 2.576855] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR3B._PRT] <7>[ 2.576886] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.BR3C._PRT] <7>[ 2.576919] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.RP08._PRT] <6>[ 2.577035] pci0000:00: Requesting ACPI _OSC control (0x15) <6>[ 2.583802] pci0000:00: ACPI _OSC control (0x15) granted <6>[ 2.590862] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 11 12 14 *15) <6>[ 2.600533] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 11 12 *14 15) <6>[ 2.610199] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 12 14 15) <6>[ 2.619927] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 *6 7 9 10 11 12 14 15) <6>[ 2.629589] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. <6>[ 2.640729] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. <6>[ 2.651822] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. <6>[ 2.662969] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 11 12 14 15) *0, disabled. <6>[ 2.674231] vgaarb: device added: PCI:0000:0a:00.0,decodes=io+mem,owns=io+mem,locks=none <6>[ 2.683787] vgaarb: loaded <6>[ 2.687098] vgaarb: bridge control possible 0000:0a:00.0 <5>[ 2.693450] SCSI subsystem initialized <7>[ 2.698018] libata version 3.00 loaded. <6>[ 2.698065] usbcore: registered new interface driver usbfs <6>[ 2.704494] usbcore: registered new interface driver hub <6>[ 2.710734] usbcore: registered new device driver usb <6>[ 2.716859] PCI: Using ACPI for IRQ routing <7>[ 2.721821] PCI: old code would have set cacheline size to 32 bytes, but clflush_size = 64 <7>[ 2.721822] PCI: pci_cache_line_size set to 64 bytes <6>[ 2.722338] NetLabel: Initializing <6>[ 2.726430] NetLabel: domain hash size = 128 <6>[ 2.731577] NetLabel: protocols = UNLABELED CIPSOv4 <6>[ 2.737418] NetLabel: unlabeled traffic allowed by default <7>[ 2.743984] alloc irq_desc for 49 on node -1 <7>[ 2.743986] alloc kstat_irqs on node -1 <7>[ 2.743989] alloc irq_2_iommu on node -1 <7>[ 2.743992] alloc irq_desc for 50 on node -1 <7>[ 2.743993] alloc kstat_irqs on node -1 <7>[ 2.743995] alloc irq_2_iommu on node -1 <7>[ 2.743997] alloc irq_desc for 51 on node -1 <7>[ 2.743998] alloc kstat_irqs on node -1 <7>[ 2.744000] alloc irq_2_iommu on node -1 <7>[ 2.744002] alloc irq_desc for 52 on node -1 <7>[ 2.744003] alloc kstat_irqs on node -1 <7>[ 2.744004] alloc irq_2_iommu on node -1 <7>[ 2.744007] alloc irq_desc for 53 on node -1 <7>[ 2.744008] alloc kstat_irqs on node -1 <7>[ 2.744010] alloc irq_2_iommu on node -1 <6>[ 2.744012] HPET: 8 timers in total, 5 timers will be used for per-cpu timer <6>[ 2.752183] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 49, 50, 51, 52, 53, 0 <6>[ 2.760603] hpet0: 8 comparators, 64-bit 14.318180 MHz counter <7>[ 2.769588] hpet: hpet2 irq 49 for MSI <7>[ 2.769918] hpet: hpet3 irq 50 for MSI <7>[ 2.770310] hpet: hpet4 irq 51 for MSI <7>[ 2.770697] hpet: hpet5 irq 52 for MSI <7>[ 2.771084] hpet: hpet6 irq 53 for MSI <6>[ 2.772243] Switching to clocksource hpet <6>[ 2.783280] pnp: PnP ACPI init <6>[ 2.786987] ACPI: bus type pnp registered <7>[ 2.791867] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active) <7>[ 2.792157] pnp 00:01: [io 0x0cf8-0x0cff] <7>[ 2.792189] pnp 00:01: Plug and Play ACPI device, IDs PNP0a08 PNP0a03 (active) <7>[ 2.792195] pnp 00:02: [mem 0xfec00000-0xfecfffff] <7>[ 2.792211] pnp 00:02: Plug and Play ACPI device, IDs PNP0003 (active) <7>[ 2.792345] pnp 00:03: [io 0x0000-0x000f] <7>[ 2.792346] pnp 00:03: [io 0x0081-0x0083] <7>[ 2.792347] pnp 00:03: [io 0x0087] <7>[ 2.792347] pnp 00:03: [io 0x0089-0x008b] <7>[ 2.792348] pnp 00:03: [io 0x008f] <7>[ 2.792349] pnp 00:03: [io 0x00c0-0x00df] <7>[ 2.792350] pnp 00:03: [dma 4] <7>[ 2.792376] pnp 00:03: Plug and Play ACPI device, IDs PNP0200 (active) <7>[ 2.792382] pnp 00:04: [io 0x0070-0x0071] <7>[ 2.792383] pnp 00:04: [io 0x0074-0x0077] <7>[ 2.792388] pnp 00:04: [irq 8] <7>[ 2.792414] pnp 00:04: Plug and Play ACPI device, IDs PNP0b00 (active) <7>[ 2.792420] pnp 00:05: [io 0x00f0] <7>[ 2.792423] pnp 00:05: [irq 13] <7>[ 2.792449] pnp 00:05: Plug and Play ACPI device, IDs PNP0c04 (active) <7>[ 2.792455] pnp 00:06: [io 0x0061] <7>[ 2.792481] pnp 00:06: Plug and Play ACPI device, IDs PNP0800 (active) <7>[ 2.792495] pnp 00:07: [mem 0xfed00000-0xfed003ff] <7>[ 2.792522] pnp 00:07: Plug and Play ACPI device, IDs PNP0103 (active) <7>[ 2.792528] pnp 00:08: [io 0x0500-0x053f] <7>[ 2.792529] pnp 00:08: [io 0x0400-0x047f] <7>[ 2.792530] pnp 00:08: [io 0x0092] <7>[ 2.792531] pnp 00:08: [io 0x0010-0x001f] <7>[ 2.792532] pnp 00:08: [io 0x0072-0x0073] <7>[ 2.792533] pnp 00:08: [io 0x0080] <7>[ 2.792534] pnp 00:08: [io 0x0084-0x0086] <7>[ 2.792535] pnp 00:08: [io 0x0088] <7>[ 2.792536] pnp 00:08: [io 0x008c-0x008e] <7>[ 2.792537] pnp 00:08: [io 0x0090-0x009f] <7>[ 2.792537] pnp 00:08: [io 0x0540-0x057f] <7>[ 2.792538] pnp 00:08: [io 0x0600-0x061f] <7>[ 2.792539] pnp 00:08: [io 0x0ca0-0x0ca5] <7>[ 2.792540] pnp 00:08: [io 0x0880-0x0883] <7>[ 2.792541] pnp 00:08: [io 0x0800-0x081f] <7>[ 2.792542] pnp 00:08: [mem 0xfeda8000-0xfedcbfff] <7>[ 2.792543] pnp 00:08: [mem 0xff000000-0xffffffff] <7>[ 2.792544] pnp 00:08: [mem 0xfee00000-0xfeefffff] <7>[ 2.792545] pnp 00:08: [mem 0xfed12000-0xfed1200f] <7>[ 2.792546] pnp 00:08: [mem 0xfed12010-0xfed1201f] <7>[ 2.792547] pnp 00:08: [mem 0xfed1b000-0xfed1bfff] <7>[ 2.792599] pnp 00:08: Plug and Play ACPI device, IDs PNP0c02 (active) <7>[ 2.792708] pnp 00:09: [io 0x03f8-0x03ff] <7>[ 2.792712] pnp 00:09: [irq 4] <7>[ 2.792763] pnp 00:09: Plug and Play ACPI device, IDs PNP0501 (active) <7>[ 2.792866] pnp 00:0a: [io 0x02f8-0x02ff] <7>[ 2.792869] pnp 00:0a: [irq 3] <7>[ 2.792918] pnp 00:0a: Plug and Play ACPI device, IDs PNP0501 (active) <7>[ 2.792927] pnp 00:0b: [io 0x0ca8] <7>[ 2.792927] pnp 00:0b: [io 0x0cac] <7>[ 2.792931] pnp 00:0b: [irq 10] <7>[ 2.792975] pnp 00:0b: Plug and Play ACPI device, IDs IPI0001 PNP0c01 (active) <6>[ 2.793029] pnp: PnP ACPI: found 12 devices <6>[ 2.797995] ACPI: ACPI bus type pnp unregistered <6>[ 2.803452] system 00:08: [io 0x0500-0x053f] has been reserved <6>[ 2.810360] system 00:08: [io 0x0400-0x047f] has been reserved <6>[ 2.817269] system 00:08: [io 0x0540-0x057f] has been reserved <6>[ 2.824177] system 00:08: [io 0x0600-0x061f] has been reserved <6>[ 2.831085] system 00:08: [io 0x0ca0-0x0ca5] has been reserved <6>[ 2.837993] system 00:08: [io 0x0880-0x0883] has been reserved <6>[ 2.844903] system 00:08: [io 0x0800-0x081f] has been reserved <6>[ 2.851812] system 00:08: [mem 0xfeda8000-0xfedcbfff] could not be reserved <6>[ 2.859889] system 00:08: [mem 0xff000000-0xffffffff] could not be reserved <6>[ 2.867965] system 00:08: [mem 0xfee00000-0xfeefffff] has been reserved <6>[ 2.875644] system 00:08: [mem 0xfed12000-0xfed1200f] has been reserved <6>[ 2.883329] system 00:08: [mem 0xfed12010-0xfed1201f] has been reserved <6>[ 2.891014] system 00:08: [mem 0xfed1b000-0xfed1bfff] has been reserved <6>[ 2.898701] system 00:0b: [io 0x0ca8] has been reserved <6>[ 2.904930] system 00:0b: [io 0x0cac] has been reserved <6>[ 2.915794] pci 0000:00:11.0: no compatible bridge window for [mem 0xffff0000-0xffffffff pref] <6>[ 2.925929] pci 0000:01:00.0: no compatible bridge window for [mem 0xfffc0000-0xffffffff pref] <6>[ 2.936070] pci 0000:01:00.1: no compatible bridge window for [mem 0xfffc0000-0xffffffff pref] <6>[ 2.946206] pci 0000:02:00.0: no compatible bridge window for [mem 0xfffc0000-0xffffffff pref] <6>[ 2.956343] pci 0000:02:00.1: no compatible bridge window for [mem 0xfffc0000-0xffffffff pref] <6>[ 2.966479] pci 0000:04:00.0: no compatible bridge window for [mem 0xfff00000-0xffffffff pref] <6>[ 2.976621] pci 0000:05:00.0: no compatible bridge window for [mem 0xfff00000-0xffffffff pref] <6>[ 2.987531] pci 0000:00:03.0: BAR 14: assigned [mem 0x92700000-0x927fffff] <6>[ 2.995510] pci 0000:00:03.1: BAR 14: assigned [mem 0x92800000-0x928fffff] <6>[ 3.003489] pci 0000:00:03.2: BAR 15: assigned [mem 0x92900000-0x929fffff pref] <6>[ 3.012170] pci 0000:00:11.0: BAR 6: assigned [mem 0x92610000-0x9261ffff pref] <6>[ 3.020750] pci 0000:00:01.0: PCI bridge to [bus 03-03] <6>[ 3.026884] pci 0000:04:00.0: BAR 6: can't assign mem pref (size 0x100000) <6>[ 3.034860] pci 0000:00:02.0: PCI bridge to [bus 04-04] <6>[ 3.040991] pci 0000:00:02.0: bridge window [mem 0x92500000-0x925fffff] <6>[ 3.048872] pci 0000:00:02.0: bridge window [mem 0x91000000-0x917fffff 64bit pref] <6>[ 3.058044] pci 0000:02:00.0: BAR 6: assigned [mem 0x92280000-0x922bffff pref] <6>[ 3.066623] pci 0000:02:00.1: BAR 6: assigned [mem 0x922c0000-0x922fffff pref] <6>[ 3.075208] pci 0000:00:03.0: PCI bridge to [bus 02-02] <6>[ 3.081343] pci 0000:00:03.0: bridge window [mem 0x92700000-0x927fffff] <6>[ 3.089223] pci 0000:00:03.0: bridge window [mem 0x92200000-0x922fffff 64bit pref] <6>[ 3.098381] pci 0000:01:00.0: BAR 6: assigned [mem 0x92380000-0x923bffff pref] <6>[ 3.106964] pci 0000:01:00.1: BAR 6: assigned [mem 0x923c0000-0x923fffff pref] <6>[ 3.115554] pci 0000:00:03.1: PCI bridge to [bus 01-01] <6>[ 3.121688] pci 0000:00:03.1: bridge window [mem 0x92800000-0x928fffff] <6>[ 3.129576] pci 0000:00:03.1: bridge window [mem 0x92300000-0x923fffff 64bit pref] <7>[ 3.138749] pci 0000:05:00.0: reg 174: [mem 0x00000000-0x0000ffff 64bit] <6>[ 3.138751] pci 0000:05:00.0: BAR 6: assigned [mem 0x92900000-0x929fffff pref] <7>[ 3.147335] pci 0000:05:00.0: reg 174: [mem 0x00000000-0x0000ffff 64bit] <6>[ 3.147336] pci 0000:05:00.0: BAR 7: can't assign mem (size 0x100000) <6>[ 3.154832] pci 0000:00:03.2: PCI bridge to [bus 05-05] <6>[ 3.160962] pci 0000:00:03.2: bridge window [io 0x2000-0x2fff] <6>[ 3.168063] pci 0000:00:03.2: bridge window [mem 0x92400000-0x924fffff] <6>[ 3.175941] pci 0000:00:03.2: bridge window [mem 0x92900000-0x929fffff pref] <6>[ 3.184524] pci 0000:00:1c.0: PCI bridge to [bus 06-06] <6>[ 3.190660] pci 0000:09:00.0: PCI bridge to [bus 0a-0a] <6>[ 3.196940] pci 0000:09:00.0: bridge window [mem 0x91800000-0x920fffff] <6>[ 3.204921] pci 0000:09:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <6>[ 3.214295] pci 0000:08:00.0: PCI bridge to [bus 09-0a] <6>[ 3.220571] pci 0000:08:00.0: bridge window [mem 0x91800000-0x920fffff] <6>[ 3.228543] pci 0000:08:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <6>[ 3.237909] pci 0000:07:00.0: PCI bridge to [bus 08-0b] <6>[ 3.244174] pci 0000:07:00.0: bridge window [mem 0x91800000-0x921fffff] <6>[ 3.252139] pci 0000:07:00.0: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <6>[ 3.261533] pci 0000:00:1c.7: PCI bridge to [bus 07-0b] <6>[ 3.267670] pci 0000:00:1c.7: bridge window [mem 0x91800000-0x921fffff] <6>[ 3.275556] pci 0000:00:1c.7: bridge window [mem 0x90000000-0x90ffffff 64bit pref] <6>[ 3.284725] Some PCI device resources are unassigned, try booting with pci=realloc <7>[ 3.293694] alloc irq_desc for 47 on node 0 <7>[ 3.293695] alloc kstat_irqs on node 0 <7>[ 3.293697] alloc irq_2_iommu on node 0 <6>[ 3.293700] pci 0000:00:01.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47 <7>[ 3.301486] pci 0000:00:01.0: setting latency timer to 64 <6>[ 3.301492] pci 0000:00:02.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47 <7>[ 3.309277] pci 0000:00:02.0: setting latency timer to 64 <6>[ 3.309282] pci 0000:00:03.0: PCI INT A -> GSI 47 (level, low) -> IRQ 47 <7>[ 3.317067] pci 0000:00:03.0: setting latency timer to 64 <6>[ 3.317072] pci 0000:00:03.1: PCI INT A -> GSI 47 (level, low) -> IRQ 47 <7>[ 3.324857] pci 0000:00:03.1: setting latency timer to 64 <6>[ 3.324862] pci 0000:00:03.2: PCI INT A -> GSI 47 (level, low) -> IRQ 47 <7>[ 3.332646] pci 0000:00:03.2: setting latency timer to 64 <7>[ 3.332651] alloc irq_desc for 16 on node 0 <7>[ 3.332652] alloc kstat_irqs on node 0 <7>[ 3.332654] alloc irq_2_iommu on node 0 <6>[ 3.332656] pci 0000:00:1c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 <7>[ 3.340440] pci 0000:00:1c.0: setting latency timer to 64 <7>[ 3.340447] alloc irq_desc for 19 on node 0 <7>[ 3.340448] alloc kstat_irqs on node 0 <7>[ 3.340449] alloc irq_2_iommu on node 0 <6>[ 3.340451] pci 0000:00:1c.7: PCI INT D -> GSI 19 (level, low) -> IRQ 19 <7>[ 3.348234] pci 0000:00:1c.7: setting latency timer to 64 <7>[ 3.348473] pci 0000:07:00.0: setting latency timer to 64 <7>[ 3.348786] pci 0000:08:00.0: setting latency timer to 64 <7>[ 3.349107] pci 0000:09:00.0: setting latency timer to 64 <7>[ 3.349151] pci_bus 0000:00: resource 4 [io 0x0000-0x03bb] <7>[ 3.349152] pci_bus 0000:00: resource 5 [io 0x03bc-0x03df] <7>[ 3.349152] pci_bus 0000:00: resource 6 [io 0x03e0-0x0cf7] <7>[ 3.349153] pci_bus 0000:00: resource 7 [io 0x1000-0xffff] <7>[ 3.349154] pci_bus 0000:00: resource 8 [mem 0x000a0000-0x000bffff] <7>[ 3.349155] pci_bus 0000:00: resource 9 [mem 0x90000000-0xfbffbfff] <7>[ 3.349156] pci_bus 0000:00: resource 10 [mem 0x30000000000-0x33fffffffff] <7>[ 3.349157] pci_bus 0000:04: resource 1 [mem 0x92500000-0x925fffff] <7>[ 3.349158] pci_bus 0000:04: resource 2 [mem 0x91000000-0x917fffff 64bit pref] <7>[ 3.349159] pci_bus 0000:02: resource 1 [mem 0x92700000-0x927fffff] <7>[ 3.349160] pci_bus 0000:02: resource 2 [mem 0x92200000-0x922fffff 64bit pref] <7>[ 3.349161] pci_bus 0000:01: resource 1 [mem 0x92800000-0x928fffff] <7>[ 3.349162] pci_bus 0000:01: resource 2 [mem 0x92300000-0x923fffff 64bit pref] <7>[ 3.349163] pci_bus 0000:05: resource 0 [io 0x2000-0x2fff] <7>[ 3.349164] pci_bus 0000:05: resource 1 [mem 0x92400000-0x924fffff] <7>[ 3.349165] pci_bus 0000:05: resource 2 [mem 0x92900000-0x929fffff pref] <7>[ 3.349166] pci_bus 0000:07: resource 1 [mem 0x91800000-0x921fffff] <7>[ 3.349167] pci_bus 0000:07: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] <7>[ 3.349168] pci_bus 0000:08: resource 1 [mem 0x91800000-0x921fffff] <7>[ 3.349169] pci_bus 0000:08: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] <7>[ 3.349170] pci_bus 0000:09: resource 1 [mem 0x91800000-0x920fffff] <7>[ 3.349171] pci_bus 0000:09: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] <7>[ 3.349172] pci_bus 0000:0a: resource 1 [mem 0x91800000-0x920fffff] <7>[ 3.349173] pci_bus 0000:0a: resource 2 [mem 0x90000000-0x90ffffff 64bit pref] <6>[ 3.349278] NET: Registered protocol family 2 <6>[ 3.354780] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes) <6>[ 3.364408] TCP established hash table entries: 524288 (order: 11, 8388608 bytes) <6>[ 3.374096] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) <6>[ 3.381979] TCP: Hash tables configured (established 524288 bind 65536) <6>[ 3.389666] TCP reno registered <6>[ 3.393869] NET: Registered protocol family 1 <7>[ 3.399051] alloc irq_desc for 18 on node 0 <7>[ 3.399052] alloc kstat_irqs on node 0 <7>[ 3.399055] alloc irq_2_iommu on node 0 <6>[ 3.399058] pci 0000:00:1a.0: PCI INT C -> GSI 18 (level, low) -> IRQ 18 <6>[ 3.423269] pci 0000:00:1a.0: PCI INT C disabled <6>[ 3.428728] pci 0000:00:1d.0: PCI INT C -> GSI 18 (level, low) -> IRQ 18 <6>[ 3.436535] pci 0000:00:1d.0: PCI INT C disabled <7>[ 3.442141] pci 0000:0a:00.0: Boot video device <6>[ 3.442201] Trying to unpack rootfs image as initramfs... <6>[ 3.590322] Freeing initrd memory: 15376k freed <6>[ 3.597097] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) <6>[ 3.604589] Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000 <6>[ 3.613754] software IO TLB at phys 0x20000000 - 0x24000000 <6>[ 3.622138] futex hash table entries: 65536 (order: 10, 4194304 bytes) <6>[ 3.636442] HugeTLB registered 2 MB page size, pre-allocated 0 pages <5>[ 3.644270] VFS: Disk quotas dquot_6.5.2 <4>[ 3.648961] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) <6>[ 3.656573] msgmni has been set to 32768 <4>[ 3.661602] ksign: Installing public key data <4>[ 3.666761] Loading keyring <4>[ 3.670179] - Added public key BBC707A16E7BE81 <4>[ 3.675436] - User ID: Red Hat, Inc. (Kernel Module GPG key) <4>[ 3.682057] - Added public key D4A26C9CCD09BEDA <4>[ 3.687413] - User ID: Red Hat Enterprise Linux Driver Update Program <4>[ 3.697269] - Added public key CB445E0DA3EA1F65 <4>[ 3.702713] - User ID: Red Hat Enterprise Linux Driver Update Program (key 4) <6>[ 3.713361] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) <6>[ 3.722278] io scheduler noop registered <6>[ 3.726954] io scheduler anticipatory registered <6>[ 3.732404] io scheduler deadline registered <6>[ 3.737494] io scheduler cfq registered (default) <7>[ 3.743158] pcieport 0000:00:01.0: setting latency timer to 64 <7>[ 3.743173] alloc irq_desc for 54 on node 0 <7>[ 3.743174] alloc kstat_irqs on node 0 <7>[ 3.743177] alloc irq_2_iommu on node 0 <7>[ 3.743183] pcieport 0000:00:01.0: irq 54 for MSI/MSI-X <7>[ 3.743233] pcieport 0000:00:02.0: setting latency timer to 64 <7>[ 3.743246] alloc irq_desc for 55 on node 0 <7>[ 3.743247] alloc kstat_irqs on node 0 <7>[ 3.743253] alloc irq_2_iommu on node 0 <7>[ 3.743257] pcieport 0000:00:02.0: irq 55 for MSI/MSI-X <7>[ 3.743300] pcieport 0000:00:03.0: setting latency timer to 64 <7>[ 3.743314] alloc irq_desc for 56 on node 0 <7>[ 3.743314] alloc kstat_irqs on node 0 <7>[ 3.743316] alloc irq_2_iommu on node 0 <7>[ 3.743319] pcieport 0000:00:03.0: irq 56 for MSI/MSI-X <7>[ 3.743360] pcieport 0000:00:03.1: setting latency timer to 64 <7>[ 3.743373] alloc irq_desc for 57 on node 0 <7>[ 3.743374] alloc kstat_irqs on node 0 <7>[ 3.743376] alloc irq_2_iommu on node 0 <7>[ 3.743379] pcieport 0000:00:03.1: irq 57 for MSI/MSI-X <7>[ 3.743419] pcieport 0000:00:03.2: setting latency timer to 64 <7>[ 3.743432] alloc irq_desc for 58 on node 0 <7>[ 3.743433] alloc kstat_irqs on node 0 <7>[ 3.743434] alloc irq_2_iommu on node 0 <7>[ 3.743438] pcieport 0000:00:03.2: irq 58 for MSI/MSI-X <7>[ 3.743480] pcieport 0000:00:1c.0: setting latency timer to 64 <7>[ 3.743504] alloc irq_desc for 59 on node 0 <7>[ 3.743504] alloc kstat_irqs on node 0 <7>[ 3.743506] alloc irq_2_iommu on node 0 <7>[ 3.743511] pcieport 0000:00:1c.0: irq 59 for MSI/MSI-X <7>[ 3.743566] pcieport 0000:00:1c.7: setting latency timer to 64 <7>[ 3.743589] alloc irq_desc for 60 on node 0 <7>[ 3.743590] alloc kstat_irqs on node 0 <7>[ 3.743591] alloc irq_2_iommu on node 0 <7>[ 3.743596] pcieport 0000:00:1c.7: irq 60 for MSI/MSI-X <6>[ 3.743704] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 <6>[ 3.750238] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 <6>[ 3.757921] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 <6>[ 3.765930] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 <6>[ 3.774714] ACPI: Power Button [PWRF] <7>[ 3.779268] ACPI: acpi_idle yielding to intel_idle <7>[ 3.780209] ACPI: \_SB_.SCK0.CP08: failed to get CPU APIC ID. <7>[ 3.780217] ACPI: \_SB_.SCK0.CP09: failed to get CPU APIC ID. <7>[ 3.780226] ACPI: \_SB_.SCK0.CP0A: failed to get CPU APIC ID. <7>[ 3.780234] ACPI: \_SB_.SCK0.CP0B: failed to get CPU APIC ID. <7>[ 3.780242] ACPI: \_SB_.SCK0.CP0C: failed to get CPU APIC ID. <7>[ 3.780251] ACPI: \_SB_.SCK0.CP0D: failed to get CPU APIC ID. <7>[ 3.780263] ACPI: \_SB_.SCK0.CP0E: failed to get CPU APIC ID. <7>[ 3.780271] ACPI: \_SB_.SCK0.CP0F: failed to get CPU APIC ID. <7>[ 3.780279] ACPI: \_SB_.SCK0.CP10: failed to get CPU APIC ID. <7>[ 3.780288] ACPI: \_SB_.SCK0.CP11: failed to get CPU APIC ID. <7>[ 3.780296] ACPI: \_SB_.SCK0.CP12: failed to get CPU APIC ID. <7>[ 3.780304] ACPI: \_SB_.SCK0.CP13: failed to get CPU APIC ID. <7>[ 3.780313] ACPI: \_SB_.SCK0.CP14: failed to get CPU APIC ID. <7>[ 3.780321] ACPI: \_SB_.SCK0.CP15: failed to get CPU APIC ID. <7>[ 3.780329] ACPI: \_SB_.SCK0.CP16: failed to get CPU APIC ID. <7>[ 3.780338] ACPI: \_SB_.SCK0.CP17: failed to get CPU APIC ID. <7>[ 3.780346] ACPI: \_SB_.SCK0.CP18: failed to get CPU APIC ID. <7>[ 3.780355] ACPI: \_SB_.SCK0.CP19: failed to get CPU APIC ID. <7>[ 3.780364] ACPI: \_SB_.SCK0.CP1A: failed to get CPU APIC ID. <7>[ 3.780372] ACPI: \_SB_.SCK0.CP1B: failed to get CPU APIC ID. <7>[ 3.780380] ACPI: \_SB_.SCK0.CP1C: failed to get CPU APIC ID. <7>[ 3.780388] ACPI: \_SB_.SCK0.CP1D: failed to get CPU APIC ID. <7>[ 3.780396] ACPI: \_SB_.SCK0.CP1E: failed to get CPU APIC ID. <7>[ 3.780404] ACPI: \_SB_.SCK0.CP1F: failed to get CPU APIC ID. <7>[ 3.780413] ACPI: \_SB_.SCK0.CP20: failed to get CPU APIC ID. <7>[ 3.780421] ACPI: \_SB_.SCK0.CP21: failed to get CPU APIC ID. <7>[ 3.780429] ACPI: \_SB_.SCK0.CP22: failed to get CPU APIC ID. <7>[ 3.780437] ACPI: \_SB_.SCK0.CP23: failed to get CPU APIC ID. <4>[ 3.783934] [Firmware Bug]: No valid trip found <6>[ 3.790515] ERST: Error Record Serialization Table (ERST) support is initialized. <6>[ 3.799567] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. <6>[ 3.809058] Non-volatile memory driver v1.3 <6>[ 3.814032] Linux agpgart interface v0.103 <6>[ 3.818982] crash memory driver: version 1.1 <6>[ 3.824050] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled <6>[ 4.075376] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A <6>[ 4.326373] serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A <6>[ 4.333572] 00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A <6>[ 4.340194] 00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A <6>[ 4.347559] brd: module loaded <6>[ 4.351691] loop: module loaded <6>[ 4.355506] input: Macintosh mouse button emulation as /devices/virtual/input/input1 <6>[ 4.364706] Fixed MDIO Bus: probed <6>[ 4.368819] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver <6>[ 4.376442] ehci_hcd 0000:00:1a.0: PCI INT C -> GSI 18 (level, low) -> IRQ 18 <7>[ 4.384738] ehci_hcd 0000:00:1a.0: setting latency timer to 64 <6>[ 4.384740] ehci_hcd 0000:00:1a.0: EHCI Host Controller <6>[ 4.390906] ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1 <6>[ 4.399701] ehci_hcd 0000:00:1a.0: debug port 2 <7>[ 4.408937] ehci_hcd 0000:00:1a.0: cache line size of 64 is not supported <6>[ 4.408950] ehci_hcd 0000:00:1a.0: irq 18, io mem 0x92603000 <6>[ 4.425258] ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00 <6>[ 4.431989] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 <6>[ 4.439862] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 <6>[ 4.448441] usb usb1: Product: EHCI Host Controller <6>[ 4.454186] usb usb1: Manufacturer: Linux 2.6.32-573.12.1.el6.atlas.x86_64 ehci_hcd <6>[ 4.463255] usb usb1: SerialNumber: 0000:00:1a.0 <6>[ 4.468726] usb usb1: configuration #1 chosen from 1 choice <6>[ 4.475263] hub 1-0:1.0: USB hub found <6>[ 4.479749] hub 1-0:1.0: 2 ports detected <6>[ 4.484566] ehci_hcd 0000:00:1d.0: PCI INT C -> GSI 18 (level, low) -> IRQ 18 <7>[ 4.492850] ehci_hcd 0000:00:1d.0: setting latency timer to 64 <6>[ 4.492852] ehci_hcd 0000:00:1d.0: EHCI Host Controller <6>[ 4.499001] ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 <6>[ 4.507791] ehci_hcd 0000:00:1d.0: debug port 2 <7>[ 4.517030] ehci_hcd 0000:00:1d.0: cache line size of 64 is not supported <6>[ 4.517036] ehci_hcd 0000:00:1d.0: irq 18, io mem 0x92602000 <6>[ 4.533255] ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00 <6>[ 4.539981] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 <6>[ 4.547863] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 <6>[ 4.556446] usb usb2: Product: EHCI Host Controller <6>[ 4.562191] usb usb2: Manufacturer: Linux 2.6.32-573.12.1.el6.atlas.x86_64 ehci_hcd <6>[ 4.571254] usb usb2: SerialNumber: 0000:00:1d.0 <6>[ 4.576734] usb usb2: configuration #1 chosen from 1 choice <6>[ 4.583268] hub 2-0:1.0: USB hub found <6>[ 4.587760] hub 2-0:1.0: 2 ports detected <6>[ 4.592564] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver <6>[ 4.599774] uhci_hcd: USB Universal Host Controller Interface driver <6>[ 4.607224] PNP: No PS/2 controller found. Probing ports directly. <6>[ 4.615312] mice: PS/2 mouse device common for all mice <6>[ 4.620259] Refined TSC clocksource calibration: 3199.997 MHz. <6>[ 4.620260] Switching to clocksource tsc <6>[ 4.632998] rtc_cmos 00:04: RTC can wake from S4 <6>[ 4.638472] rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0 <6>[ 4.645596] rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs <6>[ 4.653666] cpuidle: using governor ladder <6>[ 4.658777] cpuidle: using governor menu <6>[ 4.663455] EFI Variables Facility v0.08 2004-May-17 <6>[ 4.669467] usbcore: registered new interface driver hiddev <6>[ 4.675993] usbcore: registered new interface driver usbhid <6>[ 4.682512] usbhid: v2.6:USB HID core driver <6>[ 4.687771] GRE over IPv4 demultiplexor driver <6>[ 4.692636] TCP cubic registered <6>[ 4.697020] Initializing XFRM netlink socket <6>[ 4.702081] NET: Registered protocol family 17 <4>[ 4.707414] registered taskstats version 1 <6>[ 4.719176] rtc_cmos 00:04: setting system clock to 2016-02-24 00:18:04 UTC (1456273084) <6>[ 4.728769] Initalizing network drop monitor service <6>[ 4.734631] Freeing unused kernel memory: 1300k freed <6>[ 4.740690] Write protecting the kernel read-only data: 10240k <6>[ 4.747604] Freeing unused kernel memory: 728k freed <6>[ 4.753621] Freeing unused kernel memory: 1572k freed <6>[ 4.766229] dracut: dracut-004-388.el6 <6>[ 4.775620] udev: starting version 147 <6>[ 4.790278] usb 1-1: new high speed USB device number 2 using ehci_hcd <6>[ 4.797882] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014) <6>[ 4.805593] mlx4_core: Initializing 0000:04:00.0 <7>[ 4.811129] alloc irq_desc for 32 on node 0 <7>[ 4.811130] alloc kstat_irqs on node 0 <7>[ 4.811133] alloc irq_2_iommu on node 0 <6>[ 4.811136] mlx4_core 0000:04:00.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32 <7>[ 4.819771] mlx4_core 0000:04:00.0: setting latency timer to 64 <6>[ 4.919701] usb 1-1: New USB device found, idVendor=8087, idProduct=800a <6>[ 4.927488] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 <6>[ 4.935797] usb 1-1: configuration #1 chosen from 1 choice <6>[ 4.942466] hub 1-1:1.0: USB hub found <6>[ 4.947074] hub 1-1:1.0: 6 ports detected <6>[ 5.054275] usb 2-1: new high speed USB device number 2 using ehci_hcd <6>[ 5.175718] usb 2-1: New USB device found, idVendor=8087, idProduct=8002 <6>[ 5.183518] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 <6>[ 5.191810] usb 2-1: configuration #1 chosen from 1 choice <6>[ 5.198340] hub 2-1:1.0: USB hub found <6>[ 5.202946] hub 2-1:1.0: 8 ports detected <6>[ 5.281472] usb 1-1.6: new high speed USB device number 3 using ehci_hcd <6>[ 5.375202] usb 1-1.6: New USB device found, idVendor=413c, idProduct=a001 <6>[ 5.383187] usb 1-1.6: New USB device strings: Mfr=1, Product=2, SerialNumber=3 <6>[ 5.391859] usb 1-1.6: Product: Gadget USB HUB <6>[ 5.397113] usb 1-1.6: Manufacturer: no manufacturer <6>[ 5.402950] usb 1-1.6: SerialNumber: 0123456789 <6>[ 5.408335] usb 1-1.6: configuration #1 chosen from 1 choice <6>[ 5.415217] hub 1-1.6:1.0: USB hub found <6>[ 5.420058] hub 1-1.6:1.0: 6 ports detected <7>[ 11.599990] alloc irq_desc for 61 on node 0 <7>[ 11.599991] alloc kstat_irqs on node 0 <7>[ 11.599994] alloc irq_2_iommu on node 0 <7>[ 11.599997] mlx4_core 0000:04:00.0: irq 61 for MSI/MSI-X <7>[ 11.599997] alloc irq_desc for 62 on node 0 <7>[ 11.599998] alloc kstat_irqs on node 0 <7>[ 11.600000] alloc irq_2_iommu on node 0 <7>[ 11.600001] mlx4_core 0000:04:00.0: irq 62 for MSI/MSI-X <7>[ 11.600002] alloc irq_desc for 63 on node 0 <7>[ 11.600003] alloc kstat_irqs on node 0 <7>[ 11.600004] alloc irq_2_iommu on node 0 <7>[ 11.600006] mlx4_core 0000:04:00.0: irq 63 for MSI/MSI-X <7>[ 11.600006] alloc irq_desc for 64 on node 0 <7>[ 11.600007] alloc kstat_irqs on node 0 <7>[ 11.600008] alloc irq_2_iommu on node 0 <7>[ 11.600010] mlx4_core 0000:04:00.0: irq 64 for MSI/MSI-X <7>[ 11.600011] alloc irq_desc for 65 on node 0 <7>[ 11.600011] alloc kstat_irqs on node 0 <7>[ 11.600012] alloc irq_2_iommu on node 0 <7>[ 11.600014] mlx4_core 0000:04:00.0: irq 65 for MSI/MSI-X <7>[ 11.600015] alloc irq_desc for 66 on node 0 <7>[ 11.600015] alloc kstat_irqs on node 0 <7>[ 11.600017] alloc irq_2_iommu on node 0 <7>[ 11.600018] mlx4_core 0000:04:00.0: irq 66 for MSI/MSI-X <7>[ 11.600019] alloc irq_desc for 67 on node 0 <7>[ 11.600020] alloc kstat_irqs on node 0 <7>[ 11.600021] alloc irq_2_iommu on node 0 <7>[ 11.600022] mlx4_core 0000:04:00.0: irq 67 for MSI/MSI-X <7>[ 11.600023] alloc irq_desc for 68 on node 0 <7>[ 11.600024] alloc kstat_irqs on node 0 <7>[ 11.600025] alloc irq_2_iommu on node 0 <7>[ 11.600027] mlx4_core 0000:04:00.0: irq 68 for MSI/MSI-X <7>[ 11.600027] alloc irq_desc for 69 on node 0 <7>[ 11.600028] alloc kstat_irqs on node 0 <7>[ 11.600029] alloc irq_2_iommu on node 0 <7>[ 11.600031] mlx4_core 0000:04:00.0: irq 69 for MSI/MSI-X <7>[ 11.600032] alloc irq_desc for 70 on node 0 <7>[ 11.600032] alloc kstat_irqs on node 0 <7>[ 11.600034] alloc irq_2_iommu on node 0 <7>[ 11.600035] mlx4_core 0000:04:00.0: irq 70 for MSI/MSI-X <7>[ 11.600036] alloc irq_desc for 71 on node 0 <7>[ 11.600037] alloc kstat_irqs on node 0 <7>[ 11.600038] alloc irq_2_iommu on node 0 <7>[ 11.600039] mlx4_core 0000:04:00.0: irq 71 for MSI/MSI-X <7>[ 11.600040] alloc irq_desc for 72 on node 0 <7>[ 11.600041] alloc kstat_irqs on node 0 <7>[ 11.600042] alloc irq_2_iommu on node 0 <7>[ 11.600044] mlx4_core 0000:04:00.0: irq 72 for MSI/MSI-X <7>[ 11.600044] alloc irq_desc for 73 on node 0 <7>[ 11.600045] alloc kstat_irqs on node 0 <7>[ 11.600046] alloc irq_2_iommu on node 0 <7>[ 11.600048] mlx4_core 0000:04:00.0: irq 73 for MSI/MSI-X <6>[ 11.638870] pps_core: LinuxPPS API ver. 1 registered <6>[ 11.644710] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <6>[ 11.659840] PTP clock support registered <6>[ 11.673219] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.2-1 (Feb 2014) <6>[ 11.681568] mlx4_en 0000:04:00.0: registered PHC clock <6>[ 11.699071] tg3.c:v3.137 (May 11, 2014) <7>[ 11.703684] alloc irq_desc for 41 on node 0 <7>[ 11.703685] alloc kstat_irqs on node 0 <7>[ 11.703688] alloc irq_2_iommu on node 0 <6>[ 11.703692] tg3 0000:01:00.0: PCI INT A -> GSI 41 (level, low) -> IRQ 41 <7>[ 11.711479] tg3 0000:01:00.0: setting latency timer to 64 <6>[ 11.720866] tg3 0000:01:00.0: eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 20:47:47:7e:c5:e0 <6>[ 11.733269] tg3 0000:01:00.0: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) <6>[ 11.744886] tg3 0000:01:00.0: eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] <6>[ 11.754266] tg3 0000:01:00.0: eth0: dma_rwctrl[00000001] dma_mask[64-bit] <7>[ 11.762250] alloc irq_desc for 45 on node 0 <7>[ 11.762251] alloc kstat_irqs on node 0 <7>[ 11.762253] alloc irq_2_iommu on node 0 <6>[ 11.762256] tg3 0000:01:00.1: PCI INT B -> GSI 45 (level, low) -> IRQ 45 <7>[ 11.770063] tg3 0000:01:00.1: setting latency timer to 64 <6>[ 11.780617] tg3 0000:01:00.1: eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 20:47:47:7e:c5:e1 <6>[ 11.792994] tg3 0000:01:00.1: eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) <6>[ 11.804610] tg3 0000:01:00.1: eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] <6>[ 11.813963] tg3 0000:01:00.1: eth1: dma_rwctrl[00000001] dma_mask[64-bit] <7>[ 11.821946] alloc irq_desc for 40 on node 0 <7>[ 11.821947] alloc kstat_irqs on node 0 <7>[ 11.821949] alloc irq_2_iommu on node 0 <6>[ 11.821953] tg3 0000:02:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40 <7>[ 11.829742] tg3 0000:02:00.0: setting latency timer to 64 <6>[ 11.838499] tg3 0000:02:00.0: eth2: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 20:47:47:7e:c5:e2 <6>[ 11.850880] tg3 0000:02:00.0: eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) <6>[ 11.862510] tg3 0000:02:00.0: eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] <6>[ 11.871878] tg3 0000:02:00.0: eth2: dma_rwctrl[00000001] dma_mask[64-bit] <7>[ 11.879847] alloc irq_desc for 44 on node 0 <7>[ 11.879848] alloc kstat_irqs on node 0 <7>[ 11.879851] alloc irq_2_iommu on node 0 <6>[ 11.879854] tg3 0000:02:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44 <7>[ 11.887654] tg3 0000:02:00.1: setting latency timer to 64 <6>[ 11.896563] tg3 0000:02:00.1: eth3: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 20:47:47:7e:c5:e3 <6>[ 11.908935] tg3 0000:02:00.1: eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1]) <6>[ 11.925878] tg3 0000:02:00.1: eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1] <6>[ 11.935347] tg3 0000:02:00.1: eth3: dma_rwctrl[00000001] dma_mask[64-bit] <7>[ 12.055737] alloc irq_desc for 74 on node 0 <7>[ 12.055739] alloc kstat_irqs on node 0 <7>[ 12.055742] alloc irq_2_iommu on node 0 <7>[ 12.055745] tg3 0000:01:00.0: irq 74 for MSI/MSI-X <7>[ 12.055746] alloc irq_desc for 75 on node 0 <7>[ 12.055746] alloc kstat_irqs on node 0 <7>[ 12.055748] alloc irq_2_iommu on node 0 <7>[ 12.055750] tg3 0000:01:00.0: irq 75 for MSI/MSI-X <7>[ 12.055751] alloc irq_desc for 76 on node 0 <7>[ 12.055751] alloc kstat_irqs on node 0 <7>[ 12.055752] alloc irq_2_iommu on node 0 <7>[ 12.055754] tg3 0000:01:00.0: irq 76 for MSI/MSI-X <7>[ 12.055755] alloc irq_desc for 77 on node 0 <7>[ 12.055756] alloc kstat_irqs on node 0 <7>[ 12.055757] alloc irq_2_iommu on node 0 <7>[ 12.055759] tg3 0000:01:00.0: irq 77 for MSI/MSI-X <7>[ 12.055759] alloc irq_desc for 78 on node 0 <7>[ 12.055760] alloc kstat_irqs on node 0 <7>[ 12.055761] alloc irq_2_iommu on node 0 <7>[ 12.055763] tg3 0000:01:00.0: irq 78 for MSI/MSI-X <6>[ 15.754596] tg3 0000:01:00.0: eth0: Link is up at 1000 Mbps, full duplex <6>[ 15.762380] tg3 0000:01:00.0: eth0: Flow control is on for TX and on for RX <6>[ 15.770451] tg3 0000:01:00.0: eth0: EEE is enabled <6>[ 21.329476] RPC: Registered named UNIX socket transport module. <6>[ 21.336389] RPC: Registered udp transport module. <6>[ 21.342011] RPC: Registered tcp transport module. <6>[ 21.347581] RPC: Registered tcp NFSv4.1 backchannel transport module. <5>[ 21.372935] Slow work thread pool: Starting up <5>[ 21.378286] Slow work thread pool: Ready <5>[ 21.383015] FS-Cache: Loaded <5>[ 21.412409] NFS: Registering the id_resolver key type <5>[ 21.418358] FS-Cache: Netfs 'nfs' registered for caching <6>[ 21.459604] NET: Registered protocol family 10 <6>[ 21.465430] lo: Disabled Privacy Extensions <6>[ 21.487576] dracut: Mounted root filesystem 172.30.16.12:/images/nfsroot/hosts/atlas1-mds1 <6>[ 21.547149] dracut: Switching root <7>[ 31.485295] eth0: no IPv6 routers present <6>[ 36.782889] udev: starting version 147 <7>[ 36.838574] ahci 0000:00:11.4: version 3.0 <6>[ 36.838587] ahci 0000:00:11.4: PCI INT A -> GSI 16 (level, low) -> IRQ 16 <7>[ 36.846552] alloc irq_desc for 79 on node 0 <7>[ 36.846553] alloc kstat_irqs on node 0 <7>[ 36.846556] alloc irq_2_iommu on node 0 <7>[ 36.846563] ahci 0000:00:11.4: irq 79 for MSI/MSI-X <6>[ 36.846586] ahci: SSS flag set, parallel bus scan disabled <6>[ 36.863264] ahci 0000:00:11.4: AHCI 0001.0300 32 slots 4 ports 6 Gbps 0xf impl SATA mode <6>[ 36.872938] ahci 0000:00:11.4: flags: 64bit ncq stag led clo pio slum part ems apst <7>[ 36.882186] ahci 0000:00:11.4: setting latency timer to 64 <6>[ 36.888298] scsi0 : ahci <6>[ 36.891724] scsi1 : ahci <6>[ 36.894917] scsi2 : ahci <6>[ 36.898112] scsi3 : ahci <6>[ 36.901262] ata1: SATA max UDMA/133 abar m2048@0x92601000 port 0x92601100 irq 79 <6>[ 36.910051] ata2: SATA max UDMA/133 abar m2048@0x92601000 port 0x92601180 irq 79 <6>[ 36.918842] ata3: SATA max UDMA/133 abar m2048@0x92601000 port 0x92601200 irq 79 <6>[ 36.927634] ata4: SATA max UDMA/133 abar m2048@0x92601000 port 0x92601280 irq 79 <6>[ 36.936472] ahci 0000:00:1f.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16 <7>[ 36.944401] alloc irq_desc for 80 on node 0 <7>[ 36.944402] alloc kstat_irqs on node 0 <7>[ 36.944405] alloc irq_2_iommu on node 0 <7>[ 36.944425] ahci 0000:00:1f.2: irq 80 for MSI/MSI-X <6>[ 36.944465] ahci: SSS flag set, parallel bus scan disabled <6>[ 36.961248] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode <6>[ 36.970925] ahci 0000:00:1f.2: flags: 64bit ncq stag led clo pio slum part ems apst <7>[ 36.981183] ahci 0000:00:1f.2: setting latency timer to 64 <6>[ 36.992369] scsi4 : ahci <6>[ 36.995560] scsi5 : ahci <6>[ 36.998759] scsi6 : ahci <6>[ 37.002013] scsi7 : ahci <6>[ 37.008551] scsi8 : ahci <6>[ 37.011741] scsi9 : ahci <6>[ 37.014899] ata5: SATA max UDMA/133 abar m2048@0x92600000 port 0x92600100 irq 80 <6>[ 37.023682] ata6: SATA max UDMA/133 abar m2048@0x92600000 port 0x92600180 irq 80 <6>[ 37.025914] udev: renamed network interface eth2 to em3 <6>[ 37.038593] ata7: SATA max UDMA/133 abar m2048@0x92600000 port 0x92600200 irq 80 <6>[ 37.047375] ata8: SATA max UDMA/133 abar m2048@0x92600000 port 0x92600280 irq 80 <6>[ 37.056166] ata9: SATA max UDMA/133 abar m2048@0x92600000 port 0x92600300 irq 80 <6>[ 37.064951] ata10: SATA max UDMA/133 abar m2048@0x92600000 port 0x92600380 irq 80 <6>[ 37.082271] udev: renamed network interface eth1 to em2 <6>[ 37.178094] udev: renamed network interface eth3 to em4 <6>[ 37.250251] ata1: SATA link down (SStatus 0 SControl 300) <6>[ 37.378273] ata5: SATA link down (SStatus 0 SControl 300) <6>[ 37.563267] ata2: SATA link down (SStatus 0 SControl 300) <6>[ 37.878270] ata3: SATA link down (SStatus 0 SControl 300) <6>[ 38.193251] ata4: SATA link down (SStatus 0 SControl 300) <6>[ 38.506277] ata6: SATA link down (SStatus 0 SControl 300) <6>[ 38.817247] ata7: SATA link down (SStatus 0 SControl 300) <6>[ 39.128261] ata8: SATA link down (SStatus 0 SControl 300) <6>[ 39.439271] ata9: SATA link down (SStatus 0 SControl 300) <6>[ 39.750267] ata10: SATA link down (SStatus 0 SControl 300) <4>[ 39.783750] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042f conflicts with OpRegion 0x0000000000000428-0x000000000000042f (\GPE0) (20090903/utaddress-254) <6>[ 39.801971] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver <4>[ 39.813878] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000053f conflicts with OpRegion 0x0000000000000538-0x0000000000000538 (\GPIB) (20090903/utaddress-254) <4>[ 39.832101] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000053f conflicts with OpRegion 0x000000000000052c-0x000000000000052d (\GPIV) (20090903/utaddress-254) <6>[ 39.850342] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver <4>[ 39.862225] lpc_ich: Resource conflict(s) found affecting gpio_ich <6>[ 39.910054] EDAC MC: Ver: 2.1.0 Jan 25 2016 <6>[ 39.921025] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 <6>[ 39.927180] EDAC sbridge: Seeking for: PCI ID 8086:2fa0 <6>[ 39.933312] EDAC sbridge: Seeking for: PCI ID 8086:2ffc <6>[ 39.939442] EDAC sbridge: Seeking for: PCI ID 8086:2ffc <6>[ 39.945572] EDAC sbridge: Seeking for: PCI ID 8086:2ffd <6>[ 39.951705] EDAC sbridge: Seeking for: PCI ID 8086:2ffd <6>[ 39.957840] EDAC sbridge: Seeking for: PCI ID 8086:2f60 <6>[ 39.963969] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 <6>[ 39.970100] EDAC sbridge: Seeking for: PCI ID 8086:2fa8 <6>[ 39.976230] EDAC sbridge: Seeking for: PCI ID 8086:2f71 <6>[ 39.982363] EDAC sbridge: Seeking for: PCI ID 8086:2f71 <6>[ 39.988498] EDAC sbridge: Seeking for: PCI ID 8086:2faa <6>[ 39.994631] EDAC sbridge: Seeking for: PCI ID 8086:2faa <6>[ 40.000761] EDAC sbridge: Seeking for: PCI ID 8086:2fab <6>[ 40.006983] EDAC sbridge: Seeking for: PCI ID 8086:2fab <6>[ 40.013113] EDAC sbridge: Seeking for: PCI ID 8086:2fac <6>[ 40.019243] EDAC sbridge: Seeking for: PCI ID 8086:2fac <6>[ 40.025373] EDAC sbridge: Seeking for: PCI ID 8086:2fad <6>[ 40.031505] EDAC sbridge: Seeking for: PCI ID 8086:2fad <6>[ 40.037636] EDAC sbridge: Seeking for: PCI ID 8086:2fbd <6>[ 40.043773] EDAC sbridge: Seeking for: PCI ID 8086:2fbd <6>[ 40.049902] EDAC sbridge: Seeking for: PCI ID 8086:2f68 <6>[ 40.056034] EDAC sbridge: Seeking for: PCI ID 8086:2f68 <6>[ 40.062182] EDAC sbridge: Seeking for: PCI ID 8086:2f79 <6>[ 40.068341] EDAC sbridge: Seeking for: PCI ID 8086:2f6a <6>[ 40.074473] EDAC sbridge: Seeking for: PCI ID 8086:2f6b <6>[ 40.080604] EDAC sbridge: Seeking for: PCI ID 8086:2f6c <6>[ 40.086740] EDAC sbridge: Seeking for: PCI ID 8086:2f6d <6>[ 40.092962] EDAC MC0: Giving out device to 'sbridge_edac.c' 'Haswell Socket#0': DEV 0000:ff:12.0 <6>[ 40.103320] EDAC sbridge: Driver loaded. <6>[ 40.146415] mpt3sas version 04.100.00.00-rh loaded <7>[ 40.152178] alloc irq_desc for 42 on node 0 <7>[ 40.152179] alloc kstat_irqs on node 0 <7>[ 40.152182] alloc irq_2_iommu on node 0 <6>[ 40.152186] mpt3sas 0000:05:00.0: PCI INT A -> GSI 42 (level, low) -> IRQ 42 <7>[ 40.160372] mpt3sas 0000:05:00.0: setting latency timer to 64 <6>[ 40.160374] mpt3sas0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (264372840 kB) <4>[ 40.225292] mpt3sas0: MSI-X vectors supported: 8, no of cores: 8, max_msix_vectors: 8 <7>[ 40.234616] alloc irq_desc for 81 on node 0 <7>[ 40.234617] alloc kstat_irqs on node 0 <7>[ 40.234619] alloc irq_2_iommu on node 0 <7>[ 40.234622] mpt3sas 0000:05:00.0: irq 81 for MSI/MSI-X <7>[ 40.234623] alloc irq_desc for 82 on node 0 <7>[ 40.234624] alloc kstat_irqs on node 0 <7>[ 40.234625] alloc irq_2_iommu on node 0 <7>[ 40.234627] mpt3sas 0000:05:00.0: irq 82 for MSI/MSI-X <7>[ 40.234628] alloc irq_desc for 83 on node 0 <7>[ 40.234628] alloc kstat_irqs on node 0 <7>[ 40.234629] alloc irq_2_iommu on node 0 <7>[ 40.234631] mpt3sas 0000:05:00.0: irq 83 for MSI/MSI-X <7>[ 40.234632] alloc irq_desc for 84 on node 0 <7>[ 40.234632] alloc kstat_irqs on node 0 <7>[ 40.234633] alloc irq_2_iommu on node 0 <7>[ 40.234635] mpt3sas 0000:05:00.0: irq 84 for MSI/MSI-X <7>[ 40.234636] alloc irq_desc for 85 on node 0 <7>[ 40.234636] alloc kstat_irqs on node 0 <7>[ 40.234638] alloc irq_2_iommu on node 0 <7>[ 40.234639] mpt3sas 0000:05:00.0: irq 85 for MSI/MSI-X <7>[ 40.234640] alloc irq_desc for 86 on node 0 <7>[ 40.234641] alloc kstat_irqs on node 0 <7>[ 40.234642] alloc irq_2_iommu on node 0 <7>[ 40.234643] mpt3sas 0000:05:00.0: irq 86 for MSI/MSI-X <7>[ 40.234644] alloc irq_desc for 87 on node 0 <7>[ 40.234645] alloc kstat_irqs on node 0 <7>[ 40.234646] alloc irq_2_iommu on node 0 <7>[ 40.234648] mpt3sas 0000:05:00.0: irq 87 for MSI/MSI-X <7>[ 40.234649] alloc irq_desc for 88 on node 0 <7>[ 40.234649] alloc kstat_irqs on node 0 <7>[ 40.234650] alloc irq_2_iommu on node 0 <7>[ 40.234652] mpt3sas 0000:05:00.0: irq 88 for MSI/MSI-X <6>[ 40.234716] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 81 <6>[ 40.240649] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 82 <6>[ 40.246581] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 83 <6>[ 40.252516] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 84 <6>[ 40.258448] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 85 <6>[ 40.264382] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 86 <6>[ 40.270313] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 87 <6>[ 40.276248] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 88 <6>[ 40.282183] mpt3sas0: iomem(0x0000000092400000), mapped(0xffffc900298c0000), size(65536) <6>[ 40.291731] mpt3sas0: ioport(0x0000000000002000), size(256) <6>[ 40.406847] mpt3sas0: Allocated physical memory: size(17181 kB) <6>[ 40.413763] mpt3sas0: Current Controller Queue Depth(9699),Max Controller Queue Depth(9920) <6>[ 40.423691] mpt3sas0: Scatter Gather Elements per IO(128) <6>[ 40.475209] mpt3sas0: LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00) <6>[ 40.486123] mpt3sas0: Protocol=( <6>[ 40.489607] Initiator <6>[ 40.492511] ,Target <6>[ 40.495214] ), <6>[ 40.497533] Capabilities=( <6>[ 40.500923] TLR <6>[ 40.503243] ,EEDP <6>[ 40.505751] ,Snapshot Buffer <6>[ 40.514625] ,Diag Trace Buffer <6>[ 40.518403] ,Task Set Full <6>[ 40.521795] ,NCQ <6>[ 40.524220] ) <6>[ 40.526790] scsi10 : Fusion MPT SAS Host <6>[ 40.531595] mpt3sas0: sending port enable !! <6>[ 42.411010] mpt3sas0: host_add: handle(0x0001), sas_addr(0x544a842024dbee00), phys(8) <5>[ 42.426643] scsi 10:0:0:0: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.436317] scsi 10:0:0:0: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.448682] scsi 10:0:0:0: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.457560] scsi 10:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.469007] scsi 10:0:0:1: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.478657] scsi 10:0:0:1: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.491019] scsi 10:0:0:1: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.499891] scsi 10:0:0:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.510696] scsi 10:0:0:2: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.520353] scsi 10:0:0:2: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.532715] scsi 10:0:0:2: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.541587] scsi 10:0:0:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.552386] scsi 10:0:0:3: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.562034] scsi 10:0:0:3: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.574400] scsi 10:0:0:3: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.583271] scsi 10:0:0:3: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.594104] scsi 10:0:0:4: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.603750] scsi 10:0:0:4: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.616140] scsi 10:0:0:4: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.625010] scsi 10:0:0:4: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.635805] scsi 10:0:0:5: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.645455] scsi 10:0:0:5: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.657828] scsi 10:0:0:5: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.666695] scsi 10:0:0:5: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.677500] scsi 10:0:0:6: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.687215] scsi 10:0:0:6: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.699583] scsi 10:0:0:6: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.708464] scsi 10:0:0:6: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.719267] scsi 10:0:0:7: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.728914] scsi 10:0:0:7: SSP: handle(0x0009), sas_addr(0x50080e5434814030), phy(0), device_name(0x50080e5434814030) <6>[ 42.741286] scsi 10:0:0:7: SSP: enclosure_logical_id(0x544a842024dbee00), slot(0) <6>[ 42.750155] scsi 10:0:0:7: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.767620] scsi 10:0:1:0: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.777287] scsi 10:0:1:0: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 42.789649] scsi 10:0:1:0: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 42.798521] scsi 10:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.810133] scsi 10:0:1:1: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.819812] scsi 10:0:1:1: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 42.832270] scsi 10:0:1:1: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 42.841148] scsi 10:0:1:1: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.852058] scsi 10:0:1:2: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.861707] scsi 10:0:1:2: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 42.874076] scsi 10:0:1:2: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 42.882948] scsi 10:0:1:2: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.893784] scsi 10:0:1:3: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.903470] scsi 10:0:1:3: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 42.915849] scsi 10:0:1:3: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 42.924722] scsi 10:0:1:3: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.935526] scsi 10:0:1:4: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.945177] scsi 10:0:1:4: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 42.957570] scsi 10:0:1:4: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 42.966444] scsi 10:0:1:4: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 42.977243] scsi 10:0:1:5: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 42.986888] scsi 10:0:1:5: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 42.999256] scsi 10:0:1:5: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 43.008131] scsi 10:0:1:5: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 43.018935] scsi 10:0:1:6: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 43.028664] scsi 10:0:1:6: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 43.041033] scsi 10:0:1:6: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 43.049913] scsi 10:0:1:6: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <5>[ 43.060711] scsi 10:0:1:7: Direct-Access NETAPP INF-01-00 0820 PQ: 0 ANSI: 5 <6>[ 43.070359] scsi 10:0:1:7: SSP: handle(0x000a), sas_addr(0x50080e5434b10030), phy(4), device_name(0x50080e5434b10030) <6>[ 43.082734] scsi 10:0:1:7: SSP: enclosure_logical_id(0x544a842024dbee00), slot(4) <6>[ 43.091606] scsi 10:0:1:7: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) <6>[ 47.543250] mpt3sas0: port enable: SUCCESS <6>[ 47.560198] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 <6>[ 47.636183] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2) <6>[ 47.660077] ipmi message handler version 39.2 <6>[ 47.672245] IPMI System Interface driver. <6>[ 47.677025] ipmi_si: probing via SMBIOS <6>[ 47.681609] ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10 <6>[ 47.688637] ipmi_si: Adding SMBIOS-specified kcs state machine <6>[ 47.695574] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10 <4>[ 47.738309] IRQ 10/ipmi_si: IRQF_DISABLED is not guaranteed on shared IRQs <6>[ 47.746311] ipmi_si ipmi_si.0: Using irq 10 <4>[ 47.749503] ipmi_si ipmi_si.0: Couldn't set irq info: cc. <4>[ 47.749505] ipmi_si ipmi_si.0: Maybe ok, but ipmi might run very slowly. <6>[ 47.784403] ipmi_si ipmi_si.0: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20) <6>[ 47.794594] ipmi_si ipmi_si.0: IPMI kcs interface initialized <4>[ 47.811721] ACPI: No handler for Region [SYSI] (ffff884027a59348) [IPMI] <6>[ 47.819767] power_meter ACPI000D:00: Found ACPI power meter. <6>[ 47.835767] ACPI: WMI: Mapper loaded <6>[ 47.888128] microcode: CPU0 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 47.895048] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 47.909503] microcode: CPU1 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 47.916341] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 47.940913] microcode: CPU2 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 47.947719] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 47.959919] microcode: CPU3 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 47.966726] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 47.979570] microcode: CPU4 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 47.986380] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 47.998604] microcode: CPU5 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 48.005414] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 48.018023] microcode: CPU6 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 48.024832] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 48.037074] microcode: CPU7 sig=0x306f2, pf=0x1, revision=0x31 <6>[ 48.043883] platform microcode: firmware: requesting intel-ucode/06-3f-02 <6>[ 48.056213] Microcode Update Driver: v2.00 , Peter Oruba <6>[ 48.072277] iTCO_vendor_support: vendor-support=0 <6>[ 48.082904] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.07rh <6>[ 48.089714] iTCO_wdt: Found a Wellsburg TCO device (Version=2, TCOBASE=0x0460) <6>[ 48.103687] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) <5>[ 48.124319] sd 10:0:0:0: [sda] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124348] sd 10:0:0:1: [sdb] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124349] sd 10:0:0:1: [sdb] 4096-byte physical blocks <5>[ 48.124358] sd 10:0:0:2: [sdc] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124358] sd 10:0:0:2: [sdc] 4096-byte physical blocks <5>[ 48.124360] sd 10:0:0:3: [sdd] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124362] sd 10:0:0:3: [sdd] 4096-byte physical blocks <5>[ 48.124369] sd 10:0:0:4: [sde] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124370] sd 10:0:0:4: [sde] 4096-byte physical blocks <5>[ 48.124425] sd 10:0:0:5: [sdf] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124426] sd 10:0:0:6: [sdg] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124427] sd 10:0:0:5: [sdf] 4096-byte physical blocks <5>[ 48.124428] sd 10:0:0:6: [sdg] 4096-byte physical blocks <5>[ 48.124465] sd 10:0:0:7: [sdh] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124466] sd 10:0:0:7: [sdh] 4096-byte physical blocks <5>[ 48.124483] sd 10:0:1:0: [sdi] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124484] sd 10:0:1:0: [sdi] 4096-byte physical blocks <5>[ 48.124525] sd 10:0:1:1: [sdj] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124526] sd 10:0:1:1: [sdj] 4096-byte physical blocks <5>[ 48.124613] sd 10:0:1:2: [sdk] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124614] sd 10:0:1:2: [sdk] 4096-byte physical blocks <5>[ 48.124651] sd 10:0:1:3: [sdl] 4294967296 512-byte logical blocks: (2.19 TB/2.00 TiB) <5>[ 48.124653] sd 10:0:1:3: [sdl] 4096-byte physical blocks <5>[ 48.124667] sd 10:0:0:1: [sdb] Write Protect is off <7>[ 48.124668] sd 10:0:0:1: [sdb] Mode Sense: 83 00 10 08 <5>[ 48.124677] sd 10:0:1:4: [sdm] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124677] sd 10:0:1:4: [sdm] 4096-byte physical blocks <5>[ 48.124679] sd 10:0:0:2: [sdc] Write Protect is off <7>[ 48.124681] sd 10:0:0:2: [sdc] Mode Sense: 83 00 10 08 <5>[ 48.124682] sd 10:0:1:5: [sdn] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124683] sd 10:0:1:5: [sdn] 4096-byte physical blocks <5>[ 48.124689] sd 10:0:0:3: [sdd] Write Protect is off <7>[ 48.124690] sd 10:0:0:3: [sdd] Mode Sense: 83 00 10 08 <5>[ 48.124710] sd 10:0:1:6: [sdo] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124711] sd 10:0:1:6: [sdo] 4096-byte physical blocks <5>[ 48.124745] sd 10:0:0:4: [sde] Write Protect is off <7>[ 48.124746] sd 10:0:0:4: [sde] Mode Sense: 83 00 10 08 <5>[ 48.124812] sd 10:0:0:1: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.124833] sd 10:0:0:2: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.124835] sd 10:0:0:3: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.124847] sd 10:0:0:5: [sdf] Write Protect is off <7>[ 48.124848] sd 10:0:0:5: [sdf] Mode Sense: 83 00 10 08 <5>[ 48.124859] sd 10:0:0:6: [sdg] Write Protect is off <7>[ 48.124860] sd 10:0:0:6: [sdg] Mode Sense: 83 00 10 08 <5>[ 48.124873] sd 10:0:0:7: [sdh] Write Protect is off <7>[ 48.124874] sd 10:0:0:7: [sdh] Mode Sense: 83 00 10 08 <5>[ 48.124879] sd 10:0:0:4: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.124909] sd 10:0:1:7: [sdp] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB) <5>[ 48.124910] sd 10:0:1:7: [sdp] 4096-byte physical blocks <5>[ 48.124938] sd 10:0:0:6: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.124943] sd 10:0:0:5: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.124949] sd 10:0:1:0: [sdi] Write Protect is off <7>[ 48.124949] sd 10:0:1:0: [sdi] Mode Sense: 83 00 10 08 <5>[ 48.124953] sd 10:0:0:7: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.124984] sd 10:0:1:1: [sdj] Write Protect is off <7>[ 48.124985] sd 10:0:1:1: [sdj] Mode Sense: 83 00 10 08 <5>[ 48.125060] sd 10:0:1:2: [sdk] Write Protect is off <7>[ 48.125061] sd 10:0:1:2: [sdk] Mode Sense: 83 00 10 08 <5>[ 48.125110] sd 10:0:1:3: [sdl] Write Protect is off <7>[ 48.125112] sd 10:0:1:3: [sdl] Mode Sense: 83 00 10 08 <5>[ 48.125116] sd 10:0:1:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.125125] sd 10:0:1:5: [sdn] Write Protect is off <7>[ 48.125126] sd 10:0:1:5: [sdn] Mode Sense: 83 00 10 08 <5>[ 48.125139] sd 10:0:1:1: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.125160] sd 10:0:1:4: [sdm] Write Protect is off <7>[ 48.125161] sd 10:0:1:4: [sdm] Mode Sense: 83 00 10 08 <5>[ 48.125189] sd 10:0:1:2: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.125204] sd 10:0:1:6: [sdo] Write Protect is off <7>[ 48.125205] sd 10:0:1:6: [sdo] Mode Sense: 83 00 10 08 <5>[ 48.125230] sd 10:0:1:3: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.125232] sd 10:0:1:5: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.125264] sd 10:0:1:4: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.125294] sd 10:0:1:6: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA <6>[ 48.125412] sdd: <6>[ 48.125456] sde: <6>[ 48.125538] sdb: <6>[ 48.125560] sdc: unknown partition table <5>[ 48.136234] sd 10:0:1:7: [sdp] Write Protect is off <7>[ 48.136235] sd 10:0:1:7: [sdp] Mode Sense: 83 00 10 08 <5>[ 48.146270] sd 10:0:1:7: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA <5>[ 48.614696] sd 10:0:0:0: [sda] 4096-byte physical blocks <6>[ 48.614753] sdf: <6>[ 48.614784] sdg: unknown partition table <4>[ 48.628124] unknown partition table <6>[ 48.632416] sdh: <5>[ 48.632482] sd 10:0:0:3: [sdd] Attached SCSI disk <4>[ 48.640520] unknown partition table <5>[ 48.640533] sd 10:0:0:0: [sda] Write Protect is off <7>[ 48.640534] sd 10:0:0:0: [sda] Mode Sense: 83 00 10 08 <5>[ 48.645537] sd 10:0:0:1: [sdb] Attached SCSI disk <5>[ 48.645587] sd 10:0:0:2: [sdc] Attached SCSI disk <5>[ 48.651270] sd 10:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA <6>[ 48.671781] sdj: unknown partition table <6>[ 48.676653] sdn: <5>[ 48.677050] sd 10:0:0:7: [sdh] Attached SCSI disk <6>[ 48.684719] sdk: unknown partition table <4>[ 48.689604] unknown partition table <6>[ 48.693926] sdi: <6>[ 48.695959] sdo: <6>[ 48.698504] sdm: <5>[ 48.698802] sd 10:0:0:5: [sdf] Attached SCSI disk <5>[ 48.698807] sd 10:0:0:4: [sde] Attached SCSI disk <5>[ 48.698861] sd 10:0:0:6: [sdg] Attached SCSI disk <4>[ 48.718147] unknown partition table <4>[ 48.722453] unknown partition table <5>[ 48.722882] sd 10:0:1:0: [sdi] Attached SCSI disk <6>[ 48.732293] sdl: <5>[ 48.732674] sd 10:0:1:1: [sdj] Attached SCSI disk <4>[ 48.740385] unknown partition table <4>[ 48.744696] unknown partition table <5>[ 48.745076] sd 10:0:1:6: [sdo] Attached SCSI disk <4>[ 48.754554] unknown partition table <4>[ 48.758846] unknown partition table <5>[ 48.759202] sd 10:0:1:4: [sdm] Attached SCSI disk <4>[ 48.768697] unknown partition table <5>[ 48.769097] sd 10:0:1:2: [sdk] Attached SCSI disk <5>[ 48.769111] sd 10:0:1:5: [sdn] Attached SCSI disk <6>[ 48.784213] sdp: <6>[ 48.786315] sda: unknown partition table <5>[ 48.791906] sd 10:0:1:3: [sdl] Attached SCSI disk <4>[ 48.792144] unknown partition table <5>[ 48.801963] sd 10:0:1:7: [sdp] Attached SCSI disk <5>[ 48.807715] sd 10:0:0:0: [sda] Attached SCSI disk <6>[ 48.857146] ipmi device interface <5>[ 48.940342] sd 10:0:0:0: Attached scsi generic sg0 type 0 <5>[ 48.946704] sd 10:0:0:1: Attached scsi generic sg1 type 0 <5>[ 48.953063] sd 10:0:0:2: Attached scsi generic sg2 type 0 <5>[ 48.959417] sd 10:0:0:3: Attached scsi generic sg3 type 0 <5>[ 48.965755] sd 10:0:0:4: Attached scsi generic sg4 type 0 <5>[ 48.972084] sd 10:0:0:5: Attached scsi generic sg5 type 0 <5>[ 48.978446] sd 10:0:0:6: Attached scsi generic sg6 type 0 <5>[ 48.984789] sd 10:0:0:7: Attached scsi generic sg7 type 0 <5>[ 48.991122] sd 10:0:1:0: Attached scsi generic sg8 type 0 <5>[ 48.997479] sd 10:0:1:1: Attached scsi generic sg9 type 0 <5>[ 49.003816] sd 10:0:1:2: Attached scsi generic sg10 type 0 <5>[ 49.010261] sd 10:0:1:3: Attached scsi generic sg11 type 0 <5>[ 49.016705] sd 10:0:1:4: Attached scsi generic sg12 type 0 <5>[ 49.023137] sd 10:0:1:5: Attached scsi generic sg13 type 0 <5>[ 49.029590] sd 10:0:1:6: Attached scsi generic sg14 type 0 <5>[ 49.036024] sd 10:0:1:7: Attached scsi generic sg15 type 0 <6>[ 49.230919] device-mapper: uevent: version 1.0.3 <6>[ 49.236537] device-mapper: ioctl: 4.29.0-ioctl (2014-10-28) initialised: dm-devel@redhat.com <6>[ 49.255034] device-mapper: multipath: version 1.6.0 loaded <5>[ 49.299855] sd 10:0:0:0: rdac: LUN 0 (IOSHIP) (unowned) <5>[ 49.306143] sd 10:0:0:1: rdac: LUN 1 (IOSHIP) (owned) <5>[ 49.312251] sd 10:0:0:2: rdac: LUN 2 (IOSHIP) (unowned) <5>[ 49.318532] sd 10:0:0:3: rdac: LUN 3 (IOSHIP) (owned) <5>[ 49.324617] sd 10:0:0:4: rdac: LUN 4 (IOSHIP) (unowned) <5>[ 49.330896] sd 10:0:0:5: rdac: LUN 5 (IOSHIP) (owned) <5>[ 49.336979] sd 10:0:0:6: rdac: LUN 6 (IOSHIP) (unowned) <5>[ 49.343256] sd 10:0:0:7: rdac: LUN 7 (IOSHIP) (owned) <5>[ 49.349335] sd 10:0:1:0: rdac: LUN 0 (IOSHIP) (owned) <5>[ 49.355418] sd 10:0:1:1: rdac: LUN 1 (IOSHIP) (unowned) <5>[ 49.361695] sd 10:0:1:2: rdac: LUN 2 (IOSHIP) (owned) <5>[ 49.367778] sd 10:0:1:3: rdac: LUN 3 (IOSHIP) (unowned) <5>[ 49.374055] sd 10:0:1:4: rdac: LUN 4 (IOSHIP) (owned) <5>[ 49.380137] sd 10:0:1:5: rdac: LUN 5 (IOSHIP) (unowned) <5>[ 49.386414] sd 10:0:1:6: rdac: LUN 6 (IOSHIP) (owned) <5>[ 49.392498] sd 10:0:1:7: rdac: LUN 7 (IOSHIP) (unowned) <6>[ 49.398626] rdac: device handler registered <6>[ 49.414966] device-mapper: multipath round-robin: version 1.0.0 loaded <6>[ 50.888566] mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014) <6>[ 51.445340] ip_tables: (C) 2000-2006 Netfilter Core Team <4>[ 51.477717] nf_conntrack version 0.5.0 (16384 buckets, 65536 max) <6>[ 51.684954] 802.1Q VLAN Support v1.8 Ben Greear <6>[ 51.692896] All bugs added by David S. Miller <6>[ 51.832423] ADDRCONF(NETDEV_UP): ib0: link is not ready <6>[ 57.688492] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready <6>[ 83.949644] Fusion MPT base driver 3.04.20 <6>[ 83.954650] Copyright (c) 1999-2008 LSI Corporation <6>[ 83.966636] Fusion MPT misc device (ioctl) driver 3.04.20 <6>[ 83.973014] mptctl: Registered with Fusion MPT base driver <6>[ 83.979466] mptctl: /dev/mptctl @ (major,minor=10,220) <6>[ 84.014452] mpt2sas version 20.101.00.00 loaded <4>[ 96.679075] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[ 99.020495] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[ 135.367621] LNet: HW CPU cores: 8, npartitions: 2 <6>[ 135.385146] alg: No test for crc32 (crc32-table) <6>[ 135.390499] alg: No test for adler32 (adler32-zlib) <6>[ 135.396110] alg: No test for crc32 (crc32-pclmul) <3>[ 143.499159] LNetError: 2878:0:(o2iblnd_cb.c:2273:kiblnd_passive_connect()) Can't accept 10.36.230.231@o2ib200 on NA (ib0:0:10.36.226.117): bad dst nid 10.36.226.117@o2ib200 <6>[ 143.596814] LNet: Added LNI 10.36.226.117@o2ib [63/2560/0/180] <6>[ 144.006446] LNet: Added LNI 10.36.226.117@o2ib200 [63/2560/0/180] <6>[ 144.206241] Lustre: Lustre: Build Version: 2.5.5-g1241c21-CHANGED-2.6.32-573.12.1.el6.atlas.x86_64 <6>[ 147.031029] LDISKFS-fs (dm-7): external journal device major/minor numbers have changed <4>[ 147.205317] LDISKFS-fs (dm-7): warning: mounting fs with errors, running e2fsck is recommended <6>[ 147.316924] LDISKFS-fs (dm-7): recovery complete <6>[ 147.334016] LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. quota=on. Opts: <6>[ 147.466904] Lustre: MGC10.36.226.77@o2ib: Connection restored to MGS (at 10.36.226.77@o2ib) <3>[ 147.513908] LustreError: 137-5: atlas1-MDT0000_UUID: not available for connect from 12888@gni100 (no target). If you are running an HA pair check that the target is mounted on the other server. <3>[ 147.533891] LustreError: Skipped 4 previous similar messages <6>[ 147.689258] Lustre: atlas1-MDT0000: Not available for connect from 1476@gni100 (not set up) <6>[ 147.699109] Lustre: Skipped 16 previous similar messages <4>[ 147.732330] Lustre: atlas1-MDT0000: used disk, loading <6>[ 147.968408] Lustre: atlas1-OST0014-osc-MDT0000: Connection restored to atlas1-OST0014 (at 10.36.225.50@o2ib) <6>[ 147.979931] Lustre: Skipped 17 previous similar messages <3>[ 148.597825] LustreError: 11-0: atlas1-MDT0000-lwp-MDT0000: Communicating with 0@lo, operation mds_connect failed with -11. <6>[ 148.605323] Lustre: atlas1-MDT0000: Imperative Recovery enabled, recovery window shrunk from 1800-5400 down to 900-2700 <4>[ 148.625898] Lustre: atlas1-MDT0000: Will be in recovery for at least 15:00, or until 19415 clients reconnect <4>[ 178.623118] Lustre: 14887:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1456273227/real 1456273227] req@ffff883f81da8800 x1527013151088668/t0(0) o8->atlas1-OST0392-osc-MDT0000@10.36.225.80@o2ib:28/4 lens 400/544 e 0 to 1 dl 1456273257 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 <6>[ 241.192148] Lustre: atlas1-MDT0000: Recovery over after 1:33, of 19415 clients 19415 recovered and 0 were evicted. <6>[ 298.600083] Lustre: atlas1-MDT0000-lwp-MDT0000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[ 298.610323] Lustre: Skipped 983 previous similar messages <6>[ 465.225533] Lustre: atlas1-OST01e2-osc-MDT0000: Connection restored to atlas1-OST01e2 (at 10.36.225.80@o2ib) <6>[ 471.876231] Lustre: atlas1-OST0032-osc-MDT0000: Connection restored to atlas1-OST0032 (at 10.36.225.80@o2ib) <6>[ 471.887763] Lustre: Skipped 3 previous similar messages <6>[ 612.026233] Lustre: atlas1-OST0392-osc-MDT0000: Connection restored to atlas1-OST0392 (at 10.36.225.80@o2ib) <4>[17462.614239] Lustre: atlas1-MDT0000: haven't heard from client 0ff7ea1d-e690-e84d-600f-4b6de5487136 (at 17288@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d679c00, cur 1456290542 expire 1456289642 last 1456289190 <4>[46449.061305] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[46452.659997] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[46454.669901] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[52323.847241] Lustre: atlas1-MDT0000: Client d373128c-a4ef-dae8-bc45-7189e3998ee8 (at 660@gni3) reconnecting <4>[52325.138811] Lustre: atlas1-MDT0000: Client bc859928-3d77-e2c0-8c81-cd61368e652f (at 529@gni3) reconnecting <4>[52326.264617] Lustre: atlas1-MDT0000: Client 52480ecd-2784-f050-b4dd-f5b3dcf1f700 (at 167@gni3) reconnecting <4>[52326.275859] Lustre: Skipped 1 previous similar message <4>[52331.076497] Lustre: atlas1-MDT0000: Client 81628dde-68be-203a-73b6-9c0a6054a92a (at 647@gni3) reconnecting <4>[52331.087756] Lustre: Skipped 1 previous similar message <4>[52347.948944] Lustre: atlas1-MDT0000: Client 029a4144-3690-04ef-aa95-3b982ab30244 (at 57@gni3) reconnecting <4>[52696.659167] Lustre: atlas1-MDT0000: Client 129e6fc2-b61b-6178-17ed-1146eb806d5e (at 70@gni3) reconnecting <4>[52696.670324] Lustre: Skipped 1 previous similar message <4>[53118.156565] Lustre: atlas1-MDT0000: Client b1fb359a-e04c-16e9-10d5-26dac9b6f25c (at 454@gni3) reconnecting <4>[53118.167810] Lustre: Skipped 1 previous similar message <4>[53417.234517] Lustre: atlas1-MDT0000: Client 147deb16-a3bf-fe60-a592-99b4bde06ea4 (at 121@gni3) reconnecting <4>[53417.245762] Lustre: Skipped 1 previous similar message <4>[53481.929077] Lustre: atlas1-MDT0000: Client 8f913476-54f7-9fca-16da-838338f468a7 (at 614@gni3) reconnecting <4>[53481.940324] Lustre: Skipped 102 previous similar messages <4>[53610.062225] Lustre: atlas1-MDT0000: Client 65aea5d6-fcd7-054a-d248-943c79b3a4cb (at 675@gni3) reconnecting <4>[53610.073499] Lustre: Skipped 191 previous similar messages <4>[53866.432389] Lustre: atlas1-MDT0000: Client 359fd4b6-d3b0-b44d-9d3d-e678e7555d56 (at 587@gni3) reconnecting <4>[53866.443639] Lustre: Skipped 308 previous similar messages <4>[54505.614646] Lustre: atlas1-MDT0000: Client 795214a3-11be-f47c-16ca-3c4539a7b884 (at 568@gni3) reconnecting <4>[54505.625893] Lustre: Skipped 1123 previous similar messages <4>[54667.571931] Lustre: atlas1-MDT0000: haven't heard from client 35147f9b-2abb-bc26-e365-99823b321471 (at 688@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7f32a400, cur 1456327747 expire 1456326847 last 1456326395 <4>[54667.596741] Lustre: Skipped 1 previous similar message <4>[55106.419864] Lustre: atlas1-MDT0000: Client 66edda08-438d-f6c1-18a5-eba921a03299 (at 179@gni3) reconnecting <4>[55106.431104] Lustre: Skipped 252 previous similar messages <4>[55753.151230] Lustre: atlas1-MDT0000: Client ca76891e-851e-b34b-7505-2eda4a5d1a7c (at 509@gni3) reconnecting <4>[55753.162467] Lustre: Skipped 10 previous similar messages <4>[55759.571613] Lustre: atlas1-MDT0000: haven't heard from client c3c4b453-9f17-9169-20f9-015e8645595b (at 686@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7f2fd800, cur 1456328839 expire 1456327939 last 1456327487 <4>[56487.015145] Lustre: atlas1-MDT0000: Client 5937fcc2-d416-7e7f-a8f6-bbf238f473ac (at 254@gni3) reconnecting <4>[56487.026389] Lustre: Skipped 5 previous similar messages <4>[57183.873389] Lustre: atlas1-MDT0000: Client 46ad3c23-f544-2ab6-4282-24621aaa7c2d (at 762@gni3) reconnecting <4>[57183.884630] Lustre: Skipped 7 previous similar messages <4>[59314.475895] Lustre: atlas1-MDT0000: Client 1a4afeac-d80a-f6b7-e261-6452e5c3ccc9 (at 500@gni3) reconnecting <4>[59314.487149] Lustre: Skipped 4 previous similar messages <4>[60131.565377] Lustre: atlas1-MDT0000: haven't heard from client 663008ce-8b3a-fe7f-82b8-56c49022daed (at 687@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7edb6800, cur 1456333211 expire 1456332311 last 1456331859 <4>[60842.334831] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[60845.806710] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[61924.136002] Lustre: atlas1-MDT0000-mdc-ffff883730a24000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[61924.136188] Lustre: client wants to enable acl, but mdt not! <6>[61924.136189] Lustre: Layout lock feature supported. <6>[61924.159178] Lustre: Skipped 1 previous similar message <4>[61924.269744] Lustre: Mounted atlas1-client <4>[61925.289308] Lustre: Unmounted atlas1-client <4>[65652.560325] Lustre: atlas1-MDT0000: haven't heard from client aff8b70e-968c-7444-b64d-dbe6ff5aa1ed (at 12471@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a78d400, cur 1456338732 expire 1456337832 last 1456337380 <4>[73830.552398] Lustre: atlas1-MDT0000: haven't heard from client 0cb25cb0-02bc-401f-0ccc-854b0da9b88f (at 1403@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7926d400, cur 1456346910 expire 1456346010 last 1456345558 <4>[75242.399751] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[75245.867349] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[131713.492863] Lustre: atlas1-MDT0000: haven't heard from client 0db97d31-1052-5c5c-d01a-29c4150d250a (at 10.36.207.152@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883fa8250c00, cur 1456404793 expire 1456403893 last 1456403441 <4>[132164.492375] Lustre: atlas1-MDT0000: haven't heard from client 7b2af0c8-c5ea-6c48-6354-2412b49a127b (at 10.36.207.152@o2ib) in 1203 seconds. I think it's dead, and I am evicting it. exp ffff883fa8250800, cur 1456405244 expire 1456404344 last 1456404041 <4>[132844.029902] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[132847.529239] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[132851.224862] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[133923.840195] INFO: task mdt00_283:15501 blocked for more than 120 seconds. <3>[133923.848126] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[133923.860968] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[133923.870296] mdt00_283 D 0000000000000000 0 15501 2 0x00000000 <4>[133923.878329] ffff883f8f1ef7e0 0000000000000046 ffff88018fc55a00 0000000000000003 <4>[133923.887284] ffff883f8f1ef760 ffffffff8105e646 ffff883f8f1ef770 ffff8840265b9520 <4>[133923.896198] ffff883f8f1ef770 ffffffff8105a81d ffff883f8f1e9ad8 ffff883f8f1effd8 <4>[133923.905104] Call Trace: <4>[133923.908158] [] ? enqueue_task+0x66/0x80 <4>[133923.914620] [] ? check_preempt_curr+0x6d/0x90 <4>[133923.921683] [] schedule_timeout+0x215/0x2e0 <4>[133923.928547] [] ? autoremove_wake_function+0x16/0x40 <4>[133923.936189] [] ? __wake_up_common+0x59/0x90 <4>[133923.943031] [] wait_for_common+0x123/0x180 <4>[133923.949803] [] ? default_wake_function+0x0/0x20 <4>[133923.957041] [] ? __queue_work+0x41/0x50 <4>[133923.963495] [] wait_for_completion+0x1d/0x20 <4>[133923.970443] [] call_usermodehelper_exec+0x10c/0x120 <4>[133923.978111] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[133923.986187] [] ? groups_free+0x54/0x60 <4>[133923.992594] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[133924.001180] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] <4>[133924.009723] [] mdt_identity_get+0x17/0x40 [mdt] <4>[133924.016983] [] old_init_ucred+0xdc/0x2d0 [mdt] <4>[133924.024130] [] mdt_init_ucred_intent_getattr+0x9d/0xe0 [mdt] <4>[133924.032899] [] mdt_intent_getattr+0x1e1/0x480 [mdt] <4>[133924.040524] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[133924.048099] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[133924.055926] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[133924.064144] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[133924.071766] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[133924.078531] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[133924.086178] [] mds_regular_handle+0x15/0x20 [mdt] <4>[133924.093638] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[133924.101029] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[133924.108379] [] kthread+0x9e/0xc0 <4>[133924.114152] [] child_rip+0xa/0x20 <4>[133924.120031] [] ? kthread+0x0/0xc0 <4>[133924.125888] [] ? child_rip+0x0/0x20 <5>[134037.856491] nfs: server 172.30.16.12 not responding, still trying <3>[134044.131428] INFO: task mdt00_217:15435 blocked for more than 120 seconds. <3>[134044.139345] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134044.146487] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134044.155802] mdt00_217 D 0000000000000003 0 15435 2 0x00000000 <4>[134044.163853] ffff883f8fe9f7e0 0000000000000046 0000000000000000 0000000000000003 <4>[134044.172776] ffff883f8fe9f760 ffffffff8105e646 000079be05577981 ffff8840265b9520 <4>[134044.181672] ffff883f8fe9f770 0000000107f5e710 ffff883f8fe15068 ffff883f8fe9ffd8 <4>[134044.190565] Call Trace: <4>[134044.193655] [] ? enqueue_task+0x66/0x80 <4>[134044.200110] [] schedule_timeout+0x215/0x2e0 <4>[134044.206952] [] ? autoremove_wake_function+0x16/0x40 <4>[134044.214571] [] ? __wake_up_common+0x59/0x90 <4>[134044.221422] [] wait_for_common+0x123/0x180 <4>[134044.228282] [] ? default_wake_function+0x0/0x20 <4>[134044.235513] [] ? __queue_work+0x41/0x50 <4>[134044.241966] [] wait_for_completion+0x1d/0x20 <4>[134044.248918] [] call_usermodehelper_exec+0x10c/0x120 <4>[134044.256575] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134044.264584] [] ? __kmalloc+0x21c/0x230 <4>[134044.270959] [] ? groups_free+0x54/0x60 <4>[134044.277327] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134044.285877] [] ? __req_capsule_get+0x166/0x710 [ptlrpc] <4>[134044.293906] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] <4>[134044.302626] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134044.309894] [] mdt_init_ucred_reint+0x11e/0x350 [mdt] <4>[134044.317722] [] mdt_reint_internal+0x251/0x780 [mdt] <4>[134044.325345] [] mdt_intent_reint+0x1ee/0x410 [mdt] <4>[134044.332774] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134044.340308] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134044.348134] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134044.356340] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134044.363967] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134044.370769] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134044.378398] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134044.385851] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134044.393204] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134044.400539] [] kthread+0x9e/0xc0 <4>[134044.406341] [] child_rip+0xa/0x20 <4>[134044.412213] [] ? kthread+0x0/0xc0 <4>[134044.418099] [] ? child_rip+0x0/0x20 <3>[134044.424169] INFO: task mdt00_283:15501 blocked for more than 120 seconds. <3>[134044.432081] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134044.439219] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134044.448554] mdt00_283 D 0000000000000000 0 15501 2 0x00000000 <4>[134044.456629] ffff883f8f1ef7e0 0000000000000046 ffff88018fc55a00 0000000000000003 <4>[134044.465543] ffff883f8f1ef760 ffffffff8105e646 ffff883f8f1ef770 ffff8840265b9520 <4>[134044.474450] ffff883f8f1ef770 ffffffff8105a81d ffff883f8f1e9ad8 ffff883f8f1effd8 <4>[134044.483340] Call Trace: <4>[134044.486421] [] ? enqueue_task+0x66/0x80 <4>[134044.492929] [] ? check_preempt_curr+0x6d/0x90 <4>[134044.499964] [] schedule_timeout+0x215/0x2e0 <4>[134044.506833] [] ? autoremove_wake_function+0x16/0x40 <4>[134044.514458] [] ? __wake_up_common+0x59/0x90 <4>[134044.521326] [] wait_for_common+0x123/0x180 <4>[134044.528092] [] ? default_wake_function+0x0/0x20 <4>[134044.535348] [] ? __queue_work+0x41/0x50 <4>[134044.541801] [] wait_for_completion+0x1d/0x20 <4>[134044.548759] [] call_usermodehelper_exec+0x10c/0x120 <4>[134044.556394] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134044.564408] [] ? groups_free+0x54/0x60 <4>[134044.570781] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134044.579324] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] <4>[134044.587902] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134044.595140] [] old_init_ucred+0xdc/0x2d0 [mdt] <4>[134044.602286] [] mdt_init_ucred_intent_getattr+0x9d/0xe0 [mdt] <4>[134044.611011] [] mdt_intent_getattr+0x1e1/0x480 [mdt] <4>[134044.618628] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134044.626174] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134044.634006] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134044.642219] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134044.649851] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134044.656604] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134044.664239] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134044.671712] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134044.679097] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134044.686429] [] kthread+0x9e/0xc0 <4>[134044.692204] [] child_rip+0xa/0x20 <4>[134044.698078] [] ? kthread+0x0/0xc0 <4>[134044.703958] [] ? child_rip+0x0/0x20 <3>[134044.710039] INFO: task mdt00_390:15608 blocked for more than 120 seconds. <3>[134044.717953] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134044.725086] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134044.734383] mdt00_390 D 0000000000000000 0 15608 2 0x00000000 <4>[134044.742433] ffff883f8dc877e0 0000000000000046 ffff88018fc55a00 0000000000000003 <4>[134044.751288] ffff883f8dc87760 ffffffff8105e646 ffff883f8dc87770 ffff8840265b9520 <4>[134044.760169] ffff883f8dc87770 ffffffff8105a81d ffff883f8e3f45f8 ffff883f8dc87fd8 <4>[134044.769058] Call Trace: <4>[134044.772109] [] ? enqueue_task+0x66/0x80 <4>[134044.778567] [] ? check_preempt_curr+0x6d/0x90 <4>[134044.785637] [] schedule_timeout+0x215/0x2e0 <4>[134044.792493] [] ? autoremove_wake_function+0x16/0x40 <4>[134044.800116] [] ? __wake_up_common+0x59/0x90 <4>[134044.806959] [] wait_for_common+0x123/0x180 <4>[134044.813752] [] ? default_wake_function+0x0/0x20 <4>[134044.820990] [] ? __queue_work+0x41/0x50 <4>[134044.827458] [] wait_for_completion+0x1d/0x20 <4>[134044.834412] [] call_usermodehelper_exec+0x10c/0x120 <4>[134044.842065] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134044.850081] [] ? __kmalloc+0x21c/0x230 <4>[134044.856451] [] ? groups_free+0x54/0x60 <4>[134044.862830] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134044.871362] [] ? __req_capsule_get+0x166/0x710 [ptlrpc] <4>[134044.879392] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] <4>[134044.888112] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134044.895368] [] mdt_init_ucred_reint+0x11e/0x350 [mdt] <4>[134044.903212] [] mdt_reint_internal+0x251/0x780 [mdt] <4>[134044.910849] [] mdt_intent_reint+0x1ee/0x410 [mdt] <4>[134044.918307] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134044.925854] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134044.933705] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134044.941944] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134044.949573] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134044.956349] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134044.963989] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134044.971434] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134044.978793] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134044.991453] [] kthread+0x9e/0xc0 <4>[134044.997228] [] child_rip+0xa/0x20 <4>[134045.003113] [] ? kthread+0x0/0xc0 <4>[134045.008993] [] ? child_rip+0x0/0x20 <3>[134165.014323] INFO: task mdt00_217:15435 blocked for more than 120 seconds. <3>[134165.022249] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134165.029392] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134165.038705] mdt00_217 D 0000000000000003 0 15435 2 0x00000000 <4>[134165.046770] ffff883f8fe9f7e0 0000000000000046 0000000000000000 0000000000000003 <4>[134165.055696] ffff883f8fe9f760 ffffffff8105e646 000079be05577981 ffff8840265b9520 <4>[134165.064638] ffff883f8fe9f770 0000000107f5e710 ffff883f8fe15068 ffff883f8fe9ffd8 <4>[134165.073582] Call Trace: <4>[134165.076657] [] ? enqueue_task+0x66/0x80 <4>[134165.083149] [] schedule_timeout+0x215/0x2e0 <4>[134165.090004] [] ? autoremove_wake_function+0x16/0x40 <4>[134165.097635] [] ? __wake_up_common+0x59/0x90 <4>[134165.104500] [] wait_for_common+0x123/0x180 <4>[134165.111261] [] ? default_wake_function+0x0/0x20 <4>[134165.118524] [] ? __queue_work+0x41/0x50 <4>[134165.124998] [] wait_for_completion+0x1d/0x20 <4>[134165.131950] [] call_usermodehelper_exec+0x10c/0x120 <4>[134165.139615] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134165.147634] [] ? __kmalloc+0x21c/0x230 <4>[134165.154013] [] ? groups_free+0x54/0x60 <4>[134165.160390] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134165.168979] [] ? __req_capsule_get+0x166/0x710 [ptlrpc] <4>[134165.177031] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] <4>[134165.185758] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134165.193019] [] mdt_init_ucred_reint+0x11e/0x350 [mdt] <4>[134165.200842] [] mdt_reint_internal+0x251/0x780 [mdt] <4>[134165.208472] [] mdt_intent_reint+0x1ee/0x410 [mdt] <4>[134165.215923] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134165.223480] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134165.231309] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134165.239526] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134165.247175] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134165.253952] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134165.261588] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134165.269039] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134165.276383] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134165.283830] [] kthread+0x9e/0xc0 <4>[134165.289624] [] child_rip+0xa/0x20 <4>[134165.295515] [] ? kthread+0x0/0xc0 <4>[134165.301408] [] ? child_rip+0x0/0x20 <3>[134165.307486] INFO: task mdt00_248:15466 blocked for more than 120 seconds. <3>[134165.315418] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134165.322573] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134165.331886] mdt00_248 D 0000000000000000 0 15466 2 0x00000000 <4>[134165.339975] ffff883f8f4037e0 0000000000000046 ffff88018fc55a00 0000000000000003 <4>[134165.348906] ffff883f8f403760 ffffffff8105e646 ffff883f8f403770 ffff8840265b9520 <4>[134165.357813] ffff883f8f403770 ffffffff8105a81d ffff883f8fb525f8 ffff883f8f403fd8 <4>[134165.366726] Call Trace: <4>[134165.369795] [] ? enqueue_task+0x66/0x80 <4>[134165.376268] [] ? check_preempt_curr+0x6d/0x90 <4>[134165.383356] [] schedule_timeout+0x215/0x2e0 <4>[134165.390220] [] ? autoremove_wake_function+0x16/0x40 <4>[134165.397959] [] ? __wake_up_common+0x59/0x90 <4>[134165.404822] [] wait_for_common+0x123/0x180 <4>[134165.411587] [] ? default_wake_function+0x0/0x20 <4>[134165.418873] [] ? __queue_work+0x41/0x50 <4>[134165.425437] [] wait_for_completion+0x1d/0x20 <4>[134165.432398] [] call_usermodehelper_exec+0x10c/0x120 <4>[134165.440055] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134165.448062] [] ? groups_free+0x54/0x60 <4>[134165.454444] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134165.463038] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] <4>[134165.471570] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134165.478831] [] old_init_ucred+0xdc/0x2d0 [mdt] <4>[134165.486057] [] mdt_init_ucred_intent_getattr+0x9d/0xe0 [mdt] <4>[134165.494801] [] mdt_intent_getattr+0x1e1/0x480 [mdt] <4>[134165.502446] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134165.510010] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134165.517857] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134165.526072] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134165.533727] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134165.540504] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134165.548142] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134165.555633] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134165.562997] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134165.570348] [] kthread+0x9e/0xc0 <4>[134165.576142] [] child_rip+0xa/0x20 <4>[134165.582021] [] ? kthread+0x0/0xc0 <4>[134165.587904] [] ? child_rip+0x0/0x20 <3>[134165.593993] INFO: task mdt00_283:15501 blocked for more than 120 seconds. <3>[134165.601915] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134165.609068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134165.618403] mdt00_283 D 0000000000000000 0 15501 2 0x00000000 <4>[134165.626490] ffff883f8f1ef7e0 0000000000000046 ffff88018fc55a00 0000000000000003 <4>[134165.635430] ffff883f8f1ef760 ffffffff8105e646 ffff883f8f1ef770 ffff8840265b9520 <4>[134165.644345] ffff883f8f1ef770 ffffffff8105a81d ffff883f8f1e9ad8 ffff883f8f1effd8 <4>[134165.653271] Call Trace: <4>[134165.656333] [] ? enqueue_task+0x66/0x80 <4>[134165.662805] [] ? check_preempt_curr+0x6d/0x90 <4>[134165.669851] [] schedule_timeout+0x215/0x2e0 <4>[134165.676717] [] ? autoremove_wake_function+0x16/0x40 <4>[134165.684359] [] ? __wake_up_common+0x59/0x90 <4>[134165.691213] [] wait_for_common+0x123/0x180 <4>[134165.697990] [] ? default_wake_function+0x0/0x20 <4>[134165.705243] [] ? __queue_work+0x41/0x50 <4>[134165.711706] [] wait_for_completion+0x1d/0x20 <4>[134165.718670] [] call_usermodehelper_exec+0x10c/0x120 <4>[134165.726339] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134165.734366] [] ? groups_free+0x54/0x60 <4>[134165.740760] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134165.749308] [] ? lustre_pack_reply_v2+0x1e1/0x280 [ptlrpc] <4>[134165.757841] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134165.765074] [] old_init_ucred+0xdc/0x2d0 [mdt] <4>[134165.772240] [] mdt_init_ucred_intent_getattr+0x9d/0xe0 [mdt] <4>[134165.780964] [] mdt_intent_getattr+0x1e1/0x480 [mdt] <4>[134165.788617] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134165.796158] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134165.804027] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134165.812243] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134165.819902] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134165.826672] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134165.834311] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134165.841768] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134165.849136] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134165.856475] [] kthread+0x9e/0xc0 <4>[134165.862272] [] child_rip+0xa/0x20 <4>[134165.868152] [] ? kthread+0x0/0xc0 <4>[134165.874054] [] ? child_rip+0x0/0x20 <3>[134165.880142] INFO: task mdt00_383:15601 blocked for more than 120 seconds. <3>[134165.888065] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134165.895218] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134165.904530] mdt00_383 D 0000000000000001 0 15601 2 0x00000000 <4>[134165.912605] ffff883f8e3337e0 0000000000000046 0000000000000000 0000000000000003 <4>[134165.921532] ffff883f8e333760 ffffffff8105e646 000079e175d2aaf8 ffff8840265b9520 <4>[134165.930453] ffff883f8e333770 0000000107f839ad ffff883f8e329068 ffff883f8e333fd8 <4>[134165.939390] Call Trace: <4>[134165.942467] [] ? enqueue_task+0x66/0x80 <4>[134165.948940] [] schedule_timeout+0x215/0x2e0 <4>[134165.955806] [] ? autoremove_wake_function+0x16/0x40 <4>[134165.963445] [] ? __wake_up_common+0x59/0x90 <4>[134165.970309] [] wait_for_common+0x123/0x180 <4>[134165.977065] [] ? default_wake_function+0x0/0x20 <4>[134165.984309] [] ? __queue_work+0x41/0x50 <4>[134165.990774] [] wait_for_completion+0x1d/0x20 <4>[134165.997734] [] call_usermodehelper_exec+0x10c/0x120 <4>[134166.005383] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134166.013403] [] ? __kmalloc+0x21c/0x230 <4>[134166.019783] [] ? groups_free+0x54/0x60 <4>[134166.026167] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134166.034763] [] ? __req_capsule_get+0x166/0x710 [ptlrpc] <4>[134166.042818] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] <4>[134166.051568] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134166.058814] [] mdt_init_ucred_reint+0x11e/0x350 [mdt] <4>[134166.066654] [] mdt_reint_internal+0x251/0x780 [mdt] <4>[134166.074301] [] mdt_intent_reint+0x1ee/0x410 [mdt] <4>[134166.081762] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134166.089324] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134166.097168] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134166.105388] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134166.113032] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134166.125209] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134166.132879] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134166.140314] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134166.147677] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134166.155017] [] kthread+0x9e/0xc0 <4>[134166.160801] [] child_rip+0xa/0x20 <4>[134166.166703] [] ? kthread+0x0/0xc0 <4>[134166.172584] [] ? child_rip+0x0/0x20 <3>[134166.178669] INFO: task mdt00_390:15608 blocked for more than 120 seconds. <3>[134166.186598] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134166.193754] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134166.203069] mdt00_390 D 0000000000000000 0 15608 2 0x00000000 <4>[134166.211158] ffff883f8dc877e0 0000000000000046 ffff88018fc55a00 0000000000000003 <4>[134166.220110] ffff883f8dc87760 ffffffff8105e646 ffff883f8dc87770 ffff8840265b9520 <4>[134166.229050] ffff883f8dc87770 ffffffff8105a81d ffff883f8e3f45f8 ffff883f8dc87fd8 <4>[134166.237957] Call Trace: <4>[134166.241008] [] ? enqueue_task+0x66/0x80 <4>[134166.247466] [] ? check_preempt_curr+0x6d/0x90 <4>[134166.254524] [] schedule_timeout+0x215/0x2e0 <4>[134166.261398] [] ? autoremove_wake_function+0x16/0x40 <4>[134166.269040] [] ? __wake_up_common+0x59/0x90 <4>[134166.275893] [] wait_for_common+0x123/0x180 <4>[134166.282662] [] ? default_wake_function+0x0/0x20 <4>[134166.289900] [] ? __queue_work+0x41/0x50 <4>[134166.296366] [] wait_for_completion+0x1d/0x20 <4>[134166.303318] [] call_usermodehelper_exec+0x10c/0x120 <4>[134166.310952] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134166.318984] [] ? __kmalloc+0x21c/0x230 <4>[134166.325358] [] ? groups_free+0x54/0x60 <4>[134166.331753] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134166.340310] [] ? __req_capsule_get+0x166/0x710 [ptlrpc] <4>[134166.348369] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] <4>[134166.357096] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134166.364361] [] mdt_init_ucred_reint+0x11e/0x350 [mdt] <4>[134166.372201] [] mdt_reint_internal+0x251/0x780 [mdt] <4>[134166.379858] [] mdt_intent_reint+0x1ee/0x410 [mdt] <4>[134166.387302] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134166.394854] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134166.402716] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134166.410937] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134166.418596] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134166.425365] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134166.433013] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134166.440477] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134166.447848] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134166.455190] [] kthread+0x9e/0xc0 <4>[134166.460966] [] child_rip+0xa/0x20 <4>[134166.466849] [] ? kthread+0x0/0xc0 <4>[134166.472767] [] ? child_rip+0x0/0x20 <3>[134166.478900] INFO: task mdt01_272:16002 blocked for more than 120 seconds. <3>[134166.486860] Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <3>[134166.493988] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. <6>[134166.503307] mdt01_272 D 0000000000000005 0 16002 2 0x00000000 <4>[134166.511358] ffff883f898e77e0 0000000000000046 0000000000000000 0000000000000003 <4>[134166.520261] ffff883f898e7760 ffffffff8105e646 000079d36e894fe0 ffff8840265b9520 <4>[134166.529138] ffff883f898e7770 0000000107f74e4c ffff883f898e3ad8 ffff883f898e7fd8 <4>[134166.538025] Call Trace: <4>[134166.541098] [] ? enqueue_task+0x66/0x80 <4>[134166.547569] [] schedule_timeout+0x215/0x2e0 <4>[134166.554414] [] ? autoremove_wake_function+0x16/0x40 <4>[134166.562069] [] ? __wake_up_common+0x59/0x90 <4>[134166.568928] [] wait_for_common+0x123/0x180 <4>[134166.575684] [] ? default_wake_function+0x0/0x20 <4>[134166.582938] [] ? __queue_work+0x41/0x50 <4>[134166.589390] [] wait_for_completion+0x1d/0x20 <4>[134166.596372] [] call_usermodehelper_exec+0x10c/0x120 <4>[134166.604024] [] mdt_identity_do_upcall+0x13d/0x4c0 [mdt] <4>[134166.612033] [] ? __kmalloc+0x21c/0x230 <4>[134166.618402] [] ? groups_free+0x54/0x60 <4>[134166.624777] [] upcall_cache_get_entry+0x1b4/0x880 [libcfs] <4>[134166.633353] [] ? __req_capsule_get+0x166/0x710 [ptlrpc] <4>[134166.641388] [] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] <4>[134166.650131] [] mdt_identity_get+0x17/0x40 [mdt] <4>[134166.657406] [] mdt_init_ucred_reint+0x11e/0x350 [mdt] <4>[134166.665338] [] mdt_reint_internal+0x251/0x780 [mdt] <4>[134166.672974] [] mdt_intent_reint+0x1ee/0x410 [mdt] <4>[134166.680413] [] mdt_intent_policy+0x3be/0x7a0 [mdt] <4>[134166.687972] [] ldlm_lock_enqueue+0x135/0xa10 [ptlrpc] <4>[134166.695812] [] ldlm_handle_enqueue0+0x85f/0x18f0 [ptlrpc] <4>[134166.704014] [] ? kmem_cache_alloc_trace+0x1b3/0x1c0 <4>[134166.711674] [] mdt_enqueue+0x46/0xe0 [mdt] <4>[134166.718434] [] mdt_handle_common+0x52a/0x1470 [mdt] <4>[134166.726080] [] mds_regular_handle+0x15/0x20 [mdt] <4>[134166.733563] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[134166.740939] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[134166.748290] [] kthread+0x9e/0xc0 <4>[134166.754071] [] child_rip+0xa/0x20 <4>[134166.759952] [] ? kthread+0x0/0xc0 <4>[134166.765821] [] ? child_rip+0x0/0x20 <4>[134397.917059] Lustre: 15468:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[134397.917060] req@ffff883439eab800 x1526987175938260/t0(0) o101->964e58f9-0784-fb16-ccd2-84a61031aa73@10.36.205.201@o2ib:0/0 lens 4616/3512 e 12 to 0 dl 1456407482 ref 2 fl Interpret:/0/0 rc 0/0 <4>[134398.916087] Lustre: 15369:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[134398.916089] req@ffff8834313fd400 x1527001301440416/t0(0) o101->2b93d5ce-b7c5-1b53-0eb3-3c7401594683@3790@gni100:0/0 lens 664/3512 e 12 to 0 dl 1456407483 ref 2 fl Interpret:/0/0 rc 0/0 <4>[134398.949248] Lustre: 15369:0:(service.c:1359:ptlrpc_at_send_early_reply()) Skipped 1 previous similar message <4>[134400.916049] Lustre: 15468:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[134400.916050] req@ffff88343fe5c000 x1527000755719024/t0(0) o101->b8bd608a-c47f-f66c-739a-9194ca531cc9@83@gni100:0/0 lens 648/3512 e 12 to 0 dl 1456407485 ref 2 fl Interpret:/0/0 rc 0/0 <4>[134403.916024] Lustre: 15468:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[134403.916025] req@ffff88341dd07c00 x1527001295566620/t0(0) o101->b807f258-9a26-9381-62a4-c2468cc15e8d@822@gni100:0/0 lens 584/3384 e 12 to 0 dl 1456407488 ref 2 fl Interpret:/0/0 rc 0/0 <4>[134451.917991] Lustre: 15636:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[134451.917992] req@ffff88343054ac00 x1527001302193072/t0(0) o101->d8b3cf2b-5597-da73-fc23-83f17e54903f@2348@gni100:0/0 lens 664/3512 e 4 to 0 dl 1456407536 ref 2 fl Interpret:/0/0 rc 0/0 <4>[134451.950942] Lustre: 15636:0:(service.c:1359:ptlrpc_at_send_early_reply()) Skipped 11 previous similar messages <4>[134523.917917] Lustre: 15446:0:(service.c:1359:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply <4>[134523.917918] req@ffff88342656d400 x1526986314708700/t0(0) o101->74b7233f-1871-02ea-fa7d-7afdd823b5e8@10.36.205.206@o2ib:0/0 lens 576/3384 e 2 to 0 dl 1456407608 ref 2 fl Interpret:/0/0 rc 0/0 <5>[134532.867121] nfs: server 172.30.16.12 OK <4>[134532.957460] Lustre: 15466:0:(service.c:2108:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:4s); client may timeout. req@ffff88342656d400 x1526986314708700/t0(0) o101->74b7233f-1871-02ea-fa7d-7afdd823b5e8@10.36.205.206@o2ib:0/0 lens 576/536 e 2 to 0 dl 1456407608 ref 1 fl Complete:/0/0 rc 301/301 <4>[134532.990859] Lustre: 15466:0:(service.c:2108:ptlrpc_server_handle_request()) Skipped 17 previous similar messages <4>[147037.480417] Lustre: atlas1-MDT0000: haven't heard from client 7cc842a0-a601-0282-bb2c-50567fa3c87d (at 10.36.207.161@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3c251c00, cur 1456420117 expire 1456419217 last 1456418765 <4>[147242.351468] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[147245.830798] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[148176.476139] Lustre: atlas1-MDT0000: haven't heard from client 29c3e912-8f0d-c82b-df38-bfd4f551364b (at 10.38.146.49@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f30d31800, cur 1456421256 expire 1456420356 last 1456419904 <4>[148627.477820] Lustre: atlas1-MDT0000: haven't heard from client cf8a4668-91a5-32c7-e936-94bfe20c9959 (at 10.38.146.52@o2ib5) in 1053 seconds. I think it's dead, and I am evicting it. exp ffff883f4226c800, cur 1456421707 expire 1456420807 last 1456420654 <4>[148627.503754] Lustre: Skipped 1 previous similar message <4>[149078.475078] Lustre: atlas1-MDT0000: haven't heard from client fc0e8ba5-f7c4-db73-4a2b-2f00a8c7e3e7 (at 10.38.146.55@o2ib5) in 1055 seconds. I think it's dead, and I am evicting it. exp ffff883f3adf3800, cur 1456422158 expire 1456421258 last 1456421103 <4>[149078.500969] Lustre: Skipped 1 previous similar message <4>[149529.475621] Lustre: atlas1-MDT0000: haven't heard from client 38e03ac2-283d-0642-6edf-a40838d38f45 (at 10.38.146.56@o2ib5) in 1055 seconds. I think it's dead, and I am evicting it. exp ffff883f467f5000, cur 1456422609 expire 1456421709 last 1456421554 <4>[149529.501490] Lustre: Skipped 1 previous similar message <4>[158706.466246] Lustre: atlas1-MDT0000: haven't heard from client 9e36c709-e59d-9212-6aa9-41482816f1fc (at 13843@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7b80d800, cur 1456431786 expire 1456430886 last 1456430434 <4>[159761.916103] Lustre: atlas1-MDT0000: Client adda860e-8913-6589-8d66-8e81431bcf56 (at 596@gni3) reconnecting <4>[159761.927461] Lustre: Skipped 4 previous similar messages <4>[159776.757395] Lustre: atlas1-MDT0000: Client 22dfc1ae-41fb-fbc9-9769-be04d1908b57 (at 723@gni3) reconnecting <4>[159776.768733] Lustre: Skipped 3 previous similar messages <4>[160007.017708] Lustre: atlas1-MDT0000: Client 391db8fe-35fc-171c-33cb-3de70b6bd9ec (at 619@gni3) reconnecting <4>[160007.034348] Lustre: Skipped 21 previous similar messages <4>[160110.988830] Lustre: atlas1-MDT0000: Client febf8e48-703f-f58a-61de-bf7d37a83e9b (at 577@gni3) reconnecting <4>[160259.935639] Lustre: atlas1-MDT0000: Client 129e6fc2-b61b-6178-17ed-1146eb806d5e (at 70@gni3) reconnecting <4>[160259.946877] Lustre: Skipped 1 previous similar message <4>[160411.637171] Lustre: atlas1-MDT0000: Client d41928ef-cb5f-6b66-174c-d1f43afb8627 (at 709@gni3) reconnecting <4>[160411.648510] Lustre: Skipped 2 previous similar messages <4>[160742.995510] Lustre: atlas1-MDT0000: Client 1ced215c-2f87-17a4-7ba8-50548fdcd120 (at 359@gni3) reconnecting <4>[160743.006865] Lustre: Skipped 4 previous similar messages <4>[161367.926137] Lustre: atlas1-MDT0000: Client 9c1660fd-3a81-86f9-a2f7-35c1e7ed03c6 (at 705@gni3) reconnecting <4>[161367.937496] Lustre: Skipped 14 previous similar messages <4>[161640.818244] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[161644.224569] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[161996.755744] Lustre: atlas1-MDT0000: Client 5c62b4d9-5a15-ad4e-3f80-391e68410bf6 (at 417@gni3) reconnecting <4>[161996.767107] Lustre: Skipped 9 previous similar messages <4>[162602.572058] Lustre: atlas1-MDT0000: Client 391db8fe-35fc-171c-33cb-3de70b6bd9ec (at 619@gni3) reconnecting <4>[162602.583428] Lustre: Skipped 13 previous similar messages <4>[163239.858974] Lustre: atlas1-MDT0000: Client f5e8f637-fb06-13f0-d762-c901a87e3f6b (at 103@gni3) reconnecting <4>[163239.870317] Lustre: Skipped 10 previous similar messages <4>[163872.709553] Lustre: atlas1-MDT0000: Client d57d2ebb-fa41-afdb-5182-291f37f74ce5 (at 718@gni3) reconnecting <4>[163872.720894] Lustre: Skipped 1 previous similar message <3>[164076.272929] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds <3>[164076.284661] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.231.3@o2ib200 (100): c: 0, oc: 0, rc: 63 <4>[164534.589800] Lustre: atlas1-MDT0000: Client ff393ed6-32ef-4da5-3402-1520af4152bf (at 350@gni3) reconnecting <4>[164534.601140] Lustre: Skipped 267 previous similar messages <4>[165612.459526] Lustre: atlas1-MDT0000: haven't heard from client 327e66bd-68da-79f2-3efd-2d23061c6190 (at 569@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ec0e000, cur 1456438692 expire 1456437792 last 1456437340 <3>[166171.270816] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 22 seconds <3>[166171.282663] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.231.3@o2ib200 (122): c: 0, oc: 0, rc: 63 <4>[166541.916447] Lustre: atlas1-MDT0000: Client 391db8fe-35fc-171c-33cb-3de70b6bd9ec (at 619@gni3) reconnecting <4>[166541.927786] Lustre: Skipped 93 previous similar messages <4>[166617.058023] Lustre: atlas1-MDT0000: Client 795214a3-11be-f47c-16ca-3c4539a7b884 (at 568@gni3) reconnecting <4>[166617.069358] Lustre: Skipped 262 previous similar messages <4>[169203.270234] Lustre: atlas1-MDT0000: Client 340072ae-85c7-2939-b328-6b932734cafd (at 590@gni3) reconnecting <4>[169203.281576] Lustre: Skipped 13 previous similar messages <4>[213217.842514] Lustre: atlas1-MDT0000: Client 17c08e2a-b7fd-654d-1894-4d5900b0a0b0 (at 15296@gni100) reconnecting <4>[213217.854254] Lustre: Skipped 1 previous similar message <6>[216977.784142] Lustre: atlas1-MDT0000-mdc-ffff883f466cec00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[216977.784431] Lustre: client wants to enable acl, but mdt not! <6>[216977.784433] Lustre: Layout lock feature supported. <6>[216977.807667] Lustre: Skipped 1008 previous similar messages <4>[216978.073550] Lustre: Mounted atlas1-client <4>[218885.403940] Lustre: atlas1-MDT0000: haven't heard from client a60def67-fc99-b896-6c1f-e921e5b29136 (at 10.36.207.152@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f462cc800, cur 1456491965 expire 1456491065 last 1456490613 <4>[218885.429874] Lustre: Skipped 79 previous similar messages <4>[219259.946159] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[219266.328579] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[219269.884193] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[220050.401495] Lustre: atlas1-MDT0000: haven't heard from client 7ec895ff-6eb0-4103-b58c-ebb44e4bf0e0 (at 10.36.207.152@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4854bc00, cur 1456493130 expire 1456492230 last 1456491778 <4>[223305.997757] Lustre: Unmounted atlas1-client <4>[226689.394179] Lustre: atlas1-MDT0000: haven't heard from client 06007345-38de-5195-a428-90b2135984c9 (at 10.36.207.208@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff884020821800, cur 1456499769 expire 1456498869 last 1456498417 <4>[233648.812819] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[233652.748591] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[233977.386478] Lustre: atlas1-MDT0000: haven't heard from client fa52bff4-d5f1-ff80-1229-012cb3374c1e (at 10.36.207.208@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4514c000, cur 1456507057 expire 1456506157 last 1456505705 <4>[233977.412388] Lustre: Skipped 1 previous similar message <4>[234428.387997] Lustre: atlas1-MDT0000: haven't heard from client 3ddc3bae-d817-387a-ebe2-d5b6af600372 (at 10.36.207.162@o2ib) in 1285 seconds. I think it's dead, and I am evicting it. exp ffff883f4e127400, cur 1456507508 expire 1456506608 last 1456506223 <4>[234732.492558] Lustre: 14889:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1456506922/real 1456506922] req@ffff883e8ed3e800 x1527014198990692/t0(0) o6->atlas1-OST02ae-osc-MDT0000@10.36.225.140@o2ib:28/4 lens 664/432 e 2 to 1 dl 1456507772 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[234732.525706] Lustre: 14889:0:(client.c:1943:ptlrpc_expire_one_request()) Skipped 6 previous similar messages <4>[234732.537201] Lustre: atlas1-OST02ae-osc-MDT0000: Connection to atlas1-OST02ae (at 10.36.225.140@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[234732.556946] Lustre: atlas1-OST02ae-osc-MDT0000: Connection restored to atlas1-OST02ae (at 10.36.225.140@o2ib) <6>[234732.568586] Lustre: Skipped 1008 previous similar messages <4>[234879.386592] Lustre: atlas1-MDT0000: haven't heard from client 95587283-f1f8-dcaa-1708-4d697a26986a (at 10.36.207.162@o2ib) in 1287 seconds. I think it's dead, and I am evicting it. exp ffff883f3fc9f000, cur 1456507959 expire 1456507059 last 1456506672 <4>[234888.566369] Lustre: 14889:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1456507098/real 1456507098] req@ffff883ea6a67400 x1527014199382500/t0(0) o6->atlas1-OST0146-osc-MDT0000@10.36.225.68@o2ib:28/4 lens 664/432 e 0 to 1 dl 1456507968 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[234888.599397] Lustre: atlas1-OST0146-osc-MDT0000: Connection to atlas1-OST0146 (at 10.36.225.68@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[234888.618943] Lustre: atlas1-OST00fd-osc-MDT0000: Connection restored to atlas1-OST00fd (at 10.36.225.139@o2ib) <4>[239219.448947] Lustre: atlas1-MDT0000: Client 61104d9e-6ea8-68b1-56bf-709ca3253a1f (at 562@gni3) reconnecting <4>[239359.235085] Lustre: atlas1-MDT0000: Client 1110975e-4e55-3559-5c7d-81e98bb35581 (at 595@gni3) reconnecting <4>[239377.574214] Lustre: atlas1-MDT0000: Client 6f4e24e6-fe98-983b-484d-7459cbe25ab9 (at 719@gni3) reconnecting <4>[239389.185595] Lustre: atlas1-MDT0000: Client f9857424-ab0b-f5fa-e262-18e36c597241 (at 257@gni3) reconnecting <4>[239589.644322] Lustre: atlas1-MDT0000: Client 3ef1f284-892e-f09f-e4ae-193e97859f30 (at 97@gni3) reconnecting <4>[239684.130360] Lustre: atlas1-MDT0000: Client 45723a5a-67f1-c4db-67d3-d5efea91fd5d (at 750@gni3) reconnecting <4>[239704.832487] Lustre: atlas1-MDT0000: Client e62071e5-e871-d344-71bf-ebdcb846a7a8 (at 618@gni3) reconnecting <4>[239704.843828] Lustre: Skipped 6 previous similar messages <4>[239737.023127] Lustre: atlas1-MDT0000: Client a59be958-62fc-3e49-3350-9dadaa53d447 (at 337@gni3) reconnecting <4>[239737.034470] Lustre: Skipped 309 previous similar messages <4>[239801.129318] Lustre: atlas1-MDT0000: Client a7623c7e-0726-df53-47f2-46163cef1f21 (at 304@gni3) reconnecting <4>[239801.140659] Lustre: Skipped 304 previous similar messages <4>[239937.990270] Lustre: atlas1-MDT0000: Client 1110975e-4e55-3559-5c7d-81e98bb35581 (at 595@gni3) reconnecting <4>[239938.001623] Lustre: Skipped 100 previous similar messages <4>[240247.260155] Lustre: atlas1-MDT0000: Client 7a7bf7cc-95bf-f92e-c06a-0da2263b080f (at 413@gni3) reconnecting <4>[240247.271611] Lustre: Skipped 2 previous similar messages <3>[240382.281736] LustreError: 77213:0:(ldlm_lockd.c:2362:ldlm_cancel_handler()) ldlm_cancel from 81@gni3 arrived at 1456513461 with bad export cookie 0 <3>[240683.640928] LustreError: 15205:0:(ldlm_lockd.c:2362:ldlm_cancel_handler()) ldlm_cancel from 86@gni3 arrived at 1456513763 with bad export cookie 0 <4>[240740.381708] Lustre: atlas1-MDT0000: haven't heard from client f9857424-ab0b-f5fa-e262-18e36c597241 (at 257@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7f39d800, cur 1456513820 expire 1456512920 last 1456512468 <4>[240740.406603] Lustre: Skipped 1 previous similar message <4>[240790.002625] Lustre: atlas1-MDT0000: Client 3573ef6c-d553-8a87-dce9-2b001f5d3f0b (at 617@gni3) reconnecting <4>[240790.013971] Lustre: Skipped 6 previous similar messages <3>[240919.193155] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 12 seconds <3>[240919.204966] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.231.33@o2ib200 (112): c: 0, oc: 0, rc: 63 <4>[241017.996120] Lustre: 14894:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1456513064/real 1456513064] req@ffff883ad79ec800 x1527014231369572/t0(0) o6->atlas1-OST03d0-osc-MDT0000@10.36.225.142@o2ib:28/4 lens 664/432 e 12 to 1 dl 1456513914 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[241018.029291] Lustre: 14894:0:(client.c:1943:ptlrpc_expire_one_request()) Skipped 19 previous similar messages <4>[241018.040846] Lustre: atlas1-OST03d0-osc-MDT0000: Connection to atlas1-OST03d0 (at 10.36.225.142@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[241018.060293] Lustre: Skipped 17 previous similar messages <6>[241018.066827] Lustre: atlas1-OST03d0-osc-MDT0000: Connection restored to atlas1-OST03d0 (at 10.36.225.142@o2ib) <6>[241018.078469] Lustre: Skipped 17 previous similar messages <4>[241117.929041] Lustre: 14890:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1456513207/real 1456513207] req@ffff883d0a896000 x1527014231708708/t0(0) o6->atlas1-OST0122-osc-MDT0000@10.36.225.32@o2ib:28/4 lens 664/432 e 1 to 1 dl 1456514057 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[241117.962021] Lustre: atlas1-OST0122-osc-MDT0000: Connection to atlas1-OST0122 (at 10.36.225.32@o2ib) was lost; in progress operations using this service will wait for recovery to complete <6>[241117.986907] Lustre: atlas1-OST0122-osc-MDT0000: Connection restored to atlas1-OST0122 (at 10.36.225.32@o2ib) <3>[241267.642887] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 454@gni3 ns: mdt-atlas1-MDT0000_UUID lock: ffff883b64096b00/0xe00c223895da3671 lrc: 4/0,0 mode: PR/PR res: [0x2002c64be:0x721d:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 454@gni3 remote: 0x223c18e28fda3989 expref: 13367 pid: 16213 timeout: 4535933373 lvb_type: 0 <3>[241267.685684] LustreError: 15824:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 454@gni3) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883b64096b00/0xe00c223895da3671 lrc: 4/0,0 mode: PR/PR res: [0x2002c64be:0x721d:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 454@gni3 remote: 0x223c18e28fda3989 expref: 13309 pid: 16213 timeout: 4535933373 lvb_type: 0 <4>[241277.910892] Lustre: 14890:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1456513352/real 1456513352] req@ffff8816f336ac00 x1527014232037324/t0(0) o6->atlas1-OST033f-osc-MDT0000@10.36.225.141@o2ib:28/4 lens 664/432 e 9 to 1 dl 1456514202 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 <4>[241277.943988] Lustre: 14890:0:(client.c:1943:ptlrpc_expire_one_request()) Skipped 1 previous similar message <4>[241277.955346] Lustre: atlas1-OST033f-osc-MDT0000: Connection to atlas1-OST033f (at 10.36.225.141@o2ib) was lost; in progress operations using this service will wait for recovery to complete <4>[241277.974813] Lustre: Skipped 1 previous similar message <6>[241277.981192] Lustre: atlas1-OST033f-osc-MDT0000: Connection restored to atlas1-OST033f (at 10.36.225.141@o2ib) <6>[241277.992818] Lustre: Skipped 1 previous similar message <3>[241305.751921] LustreError: 78393:0:(ldlm_lockd.c:2362:ldlm_cancel_handler()) ldlm_cancel from 454@gni3 arrived at 1456514385 with bad export cookie 16144316330451459416 <4>[241413.394579] Lustre: atlas1-MDT0000: Client e18a7e9d-630e-4811-7275-a9040d867010 (at 418@gni3) reconnecting <4>[241413.405926] Lustre: Skipped 68 previous similar messages <4>[242015.482788] Lustre: atlas1-MDT0000: Client 51121741-ffe1-c334-d990-88dbfeba2cdd (at 613@gni3) reconnecting <4>[242015.494134] Lustre: Skipped 446 previous similar messages <4>[242140.380833] Lustre: atlas1-MDT0000: haven't heard from client 3573ef6c-d553-8a87-dce9-2b001f5d3f0b (at 617@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ec5fc00, cur 1456515220 expire 1456514320 last 1456513868 <4>[242140.405736] Lustre: Skipped 1 previous similar message <3>[242300.192046] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 16 seconds <3>[242300.203871] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.231.3@o2ib200 (116): c: 0, oc: 0, rc: 63 <4>[242619.875339] Lustre: atlas1-MDT0000: Client 53ef3554-15fc-8899-6a4a-2387d8f1356d (at 84@gni3) reconnecting <4>[242619.886584] Lustre: Skipped 235 previous similar messages <4>[244626.776885] Lustre: atlas1-MDT0000: Client 381e808b-d766-3cdf-37d7-a78b4305413f (at 395@gni3) reconnecting <4>[244626.788235] Lustre: Skipped 66 previous similar messages <3>[244749.190853] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 24 seconds <3>[244749.202688] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.231.3@o2ib200 (124): c: 0, oc: 0, rc: 63 <4>[244755.015596] Lustre: atlas1-MDT0000: Client 20cd796d-26f2-4ed2-c3c2-5db6fd02704d (at 714@gni3) reconnecting <4>[244755.026978] Lustre: Skipped 79 previous similar messages <4>[245001.834937] Lustre: atlas1-MDT0000: Client a661b8df-ca3b-cd0d-ee7e-a1b35bf8aeb8 (at 7@gni3) reconnecting <4>[245001.846124] Lustre: Skipped 229 previous similar messages <4>[245302.925099] Lustre: atlas1-MDT0000: Client ca76891e-851e-b34b-7505-2eda4a5d1a7c (at 509@gni3) reconnecting <4>[245302.936460] Lustre: Skipped 26 previous similar messages <4>[245962.071370] Lustre: atlas1-MDT0000: Client 53cbd8ae-aedd-0bf3-ef42-9478b9860022 (at 517@gni3) reconnecting <4>[245962.082722] Lustre: Skipped 46 previous similar messages <4>[246617.177223] Lustre: atlas1-MDT0000: Client 7112ce29-a992-04b4-46b5-b52457e21735 (at 25@gni3) reconnecting <4>[246617.188468] Lustre: Skipped 5 previous similar messages <4>[248043.718996] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[248047.362752] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[248472.375105] Lustre: atlas1-MDT0000: haven't heard from client f152d2fb-01ac-ef2d-ed8a-53e16cf9394b (at 597@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7f2c9000, cur 1456521552 expire 1456520652 last 1456520200 <4>[248472.399992] Lustre: Skipped 1 previous similar message <4>[260624.079217] Lustre: atlas1-MDT0000: Client 62899fd8-d695-97b4-377d-20ac3406a73e (at 3337@gni100) reconnecting <4>[304713.322535] Lustre: atlas1-MDT0000: haven't heard from client 44432070-4961-f5bd-f95b-ff1dddb5c150 (at 18680@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a353400, cur 1456577793 expire 1456576893 last 1456576441 <4>[304713.347819] Lustre: Skipped 734 previous similar messages <4>[305645.269742] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[305648.904747] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[320041.737810] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[320045.385253] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[334446.301744] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[334450.094201] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[334452.228414] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[358562.271944] Lustre: atlas1-MDT0000: haven't heard from client 6b27a611-cca9-d8dc-cc6b-0edab0e75b11 (at 103@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c019800, cur 1456631642 expire 1456630742 last 1456630290 <4>[358562.297053] Lustre: Skipped 1 previous similar message <4>[375950.258072] Lustre: atlas1-MDT0000: haven't heard from client 905629cc-8a22-2720-e3f9-8cf0a28a6587 (at 6789@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f78001800, cur 1456649030 expire 1456648130 last 1456647678 <4>[376401.257799] Lustre: atlas1-MDT0000: haven't heard from client af83204e-dc16-d01c-3d99-0ffcde66a9df (at 1543@gni100) in 1347 seconds. I think it's dead, and I am evicting it. exp ffff883f7b1ac000, cur 1456649481 expire 1456648581 last 1456648134 <4>[376401.282980] Lustre: Skipped 98 previous similar messages <4>[378627.256299] Lustre: atlas1-MDT0000: haven't heard from client dd0369e6-a632-9ab6-abd8-e6dcabc562b3 (at 11125@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f79407400, cur 1456651707 expire 1456650807 last 1456650355 <4>[378627.281575] Lustre: Skipped 17 previous similar messages <4>[379964.255277] Lustre: atlas1-MDT0000: haven't heard from client b556acc6-27c6-f706-1e5c-cdfd2ceb89ae (at 13843@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3bd82c00, cur 1456653044 expire 1456652144 last 1456651692 <6>[389524.106992] Lustre: atlas1-MDT0000-mdc-ffff883f417e7c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[389524.107150] Lustre: client wants to enable acl, but mdt not! <6>[389524.107152] Lustre: Layout lock feature supported. <4>[389524.246422] Lustre: Mounted atlas1-client <4>[392045.777300] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[392049.700277] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[392052.010964] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[393819.689729] Lustre: Unmounted atlas1-client <4>[406442.127761] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[406445.728916] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[420841.315289] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[420844.973412] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[420846.992954] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[441304.195052] Lustre: atlas1-MDT0000: haven't heard from client 97a4a786-8842-3a5e-5ee4-3a526225b01f (at 10.38.144.247@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f432a2400, cur 1456714384 expire 1456713484 last 1456713032 <4>[441304.221023] Lustre: Skipped 1 previous similar message <4>[442123.982360] Lustre: atlas1-MDT0000: Client e0a4ac49-3080-d559-8bd6-c9bb5765c095 (at 10342@gni100) reconnecting <4>[442134.890083] Lustre: atlas1-MDT0000: Client 2be89de0-54f0-b976-d522-959d95904602 (at 10406@gni100) reconnecting <4>[442134.901813] Lustre: Skipped 37 previous similar messages <4>[442423.202852] Lustre: atlas1-MDT0000: Client 300835bd-b798-9cf4-f228-85600a2b1189 (at 11942@gni100) reconnecting <4>[442423.214670] Lustre: Skipped 10 previous similar messages <4>[442690.274177] Lustre: atlas1-MDT0000: Client ad4c7b7d-2fca-217c-dc5d-cc46da40044d (at 12606@gni100) reconnecting <4>[442690.285908] Lustre: Skipped 16 previous similar messages <4>[463743.176397] Lustre: atlas1-MDT0000: haven't heard from client 00cfde73-28e1-5430-5618-c8147bd70b81 (at 12963@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ccfbc00, cur 1456736823 expire 1456735923 last 1456735471 <4>[463743.201688] Lustre: Skipped 3 previous similar messages <4>[470921.167392] Lustre: atlas1-MDT0000: haven't heard from client 658eee59-fb99-e4a2-e0c0-08ea2c7c4887 (at 18662@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d7ce000, cur 1456744001 expire 1456743101 last 1456742649 <4>[478450.876673] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[478455.480013] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[478458.401115] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[483306.155891] Lustre: atlas1-MDT0000: haven't heard from client 1f6f7036-5671-e760-2235-6a049f1fc23c (at 10.38.145.143@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3a2a6c00, cur 1456756386 expire 1456755486 last 1456755034 <4>[483306.181867] Lustre: Skipped 9 previous similar messages <4>[492842.727683] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[492847.222324] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[492849.452177] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[493924.631395] Lustre: atlas1-MDT0000-mdc-ffff883f36a63c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[493924.631588] Lustre: client wants to enable acl, but mdt not! <6>[493924.631589] Lustre: Layout lock feature supported. <6>[493924.654861] Lustre: Skipped 1008 previous similar messages <4>[493924.771774] Lustre: Mounted atlas1-client <4>[499362.734579] Lustre: Unmounted atlas1-client <4>[501368.250620] Lustre: atlas1-MDT0000: Client b23bff30-d25c-3ff7-347d-908c6005c877 (at 10.36.205.200@o2ib) reconnecting <4>[501368.262931] Lustre: Skipped 482 previous similar messages <4>[502664.095112] Lustre: atlas1-MDT0000: Client 304ef19f-7a2c-8517-99f5-d9a7a06ac80d (at 3089@gni100) reconnecting <4>[507258.235193] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[507262.905398] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[507266.397813] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[512539.127353] Lustre: atlas1-MDT0000: haven't heard from client 55eea33b-c2f7-9a35-5e36-7a93016102dd (at 8231@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f780b9400, cur 1456785619 expire 1456784719 last 1456784267 <4>[512990.127093] Lustre: atlas1-MDT0000: haven't heard from client 2c4f6653-ffa0-ad01-1197-f18c77e897d3 (at 10504@gni100) in 1350 seconds. I think it's dead, and I am evicting it. exp ffff883f789bfc00, cur 1456786070 expire 1456785170 last 1456784720 <4>[512990.152369] Lustre: Skipped 109 previous similar messages <4>[564852.868738] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[564856.437184] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[568009.339843] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9311@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8824072da100/0xe00c224aded52e2d lrc: 4/0,0 mode: PR/PR res: [0x2002c772e:0xb3b0:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x60200000000020 nid: 9311@gni100 remote: 0x5ad4d3a5f14a4241 expref: 176921 pid: 16034 timeout: 4862675023 lvb_type: 0 <3>[568009.383226] LustreError: 16165:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9311@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff881eff58cd80/0xe00c2248af6b1d95 lrc: 4/0,0 mode: CR/CR res: [0x2002c6b54:0x1ce90:0x0].0 bits 0x9 rrc: 3 type: IBT flags: 0x60200000000020 nid: 9311@gni100 remote: 0x5ad4d3a5f0b51295 expref: 176918 pid: 15852 timeout: 4863040510 lvb_type: 0 <3>[568009.425101] LustreError: 16165:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 2 previous similar messages <4>[579240.380161] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[579244.031470] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[591588.057530] Lustre: atlas1-MDT0000: haven't heard from client 4cc2e20f-5d1f-2785-f326-ac2cc30f1bd7 (at 12963@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f81441400, cur 1456864668 expire 1456863768 last 1456863316 <4>[591588.082880] Lustre: Skipped 21 previous similar messages <4>[593640.610727] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[593644.321455] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[593646.334331] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[627042.001544] Lustre: atlas1-MDT0000: Client 7841344e-17c4-2338-f807-bdebe2e6afc0 (at 12257@gni100) reconnecting <6>[648723.843589] Lustre: atlas1-MDT0000-mdc-ffff883f81a87800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[648723.843749] Lustre: client wants to enable acl, but mdt not! <6>[648723.843751] Lustre: Layout lock feature supported. <6>[648723.867079] Lustre: Skipped 1008 previous similar messages <4>[648723.980172] Lustre: Mounted atlas1-client <4>[651243.700135] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[651247.330858] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[653029.099358] Lustre: Unmounted atlas1-client <4>[654435.998438] Lustre: atlas1-MDT0000: haven't heard from client e2a5277b-7ee6-afc0-8654-3950cc1197d4 (at 19073@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a24f400, cur 1456927516 expire 1456926616 last 1456926164 <4>[654436.023744] Lustre: Skipped 1 previous similar message <4>[656647.998532] Lustre: atlas1-MDT0000: haven't heard from client 5d0b47e7-d24a-fd2c-8a8c-acc0f5c9f4bb (at 2605@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f78225c00, cur 1456929728 expire 1456928828 last 1456928376 <4>[657098.995764] Lustre: atlas1-MDT0000: haven't heard from client 8b7e70ed-dcf8-4a4a-2687-b8878e1e6599 (at 3531@gni100) in 1348 seconds. I think it's dead, and I am evicting it. exp ffff883f7a3cb800, cur 1456930179 expire 1456929279 last 1456928831 <4>[657099.020966] Lustre: Skipped 92 previous similar messages <4>[664918.001009] Lustre: atlas1-MDT0000: Client bffc77c6-17ec-1282-dc92-e916ba71d980 (at 2443@gni100) reconnecting <4>[665003.413448] Lustre: atlas1-MDT0000: Client aef67553-c099-c9d8-c123-4035d66b6ce5 (at 1077@gni100) reconnecting <4>[665047.497818] Lustre: atlas1-MDT0000: Client aef67553-c099-c9d8-c123-4035d66b6ce5 (at 1077@gni100) reconnecting <4>[665644.584843] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[665648.335078] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[665650.399656] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[666398.986614] Lustre: atlas1-MDT0000: haven't heard from client aef67553-c099-c9d8-c123-4035d66b6ce5 (at 1077@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a656800, cur 1456939479 expire 1456938579 last 1456938127 <4>[666399.011792] Lustre: Skipped 17 previous similar messages <4>[680044.131559] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[680047.767242] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[680049.799622] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[700685.956000] Lustre: atlas1-MDT0000: haven't heard from client 83c9a01e-6188-a837-116b-b74fced06e11 (at 10.36.247.137@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3c251000, cur 1456973766 expire 1456972866 last 1456972414 <4>[706678.362456] Lustre: atlas1-MDT0000: Client 71e68641-32d3-a311-700c-6e61857b2cb2 (at 11694@gni100) reconnecting <4>[733083.391029] Lustre: atlas1-MDT0000: Client 0f6f387b-5987-6c6d-805d-82b588d5017c (at 11557@gni100) reconnecting <4>[733084.497062] Lustre: atlas1-MDT0000: Client 3a24eaef-3667-09e8-6353-51e921678a39 (at 14649@gni100) reconnecting <4>[733533.261137] Lustre: atlas1-MDT0000: Client 39ae2550-9712-4815-1666-b74e2f508cd0 (at 11583@gni100) reconnecting <4>[737645.246272] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[737648.899582] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[737650.922229] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[752041.851181] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[752045.470969] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[752047.492281] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[766440.476422] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[766444.404649] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[766446.594257] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[792938.868803] Lustre: atlas1-MDT0000: haven't heard from client 05b532ea-554a-71dd-8fde-a3034cafa547 (at 16841@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f798a4800, cur 1457066019 expire 1457065119 last 1457064667 <4>[795765.864921] Lustre: atlas1-MDT0000: haven't heard from client ed5b21dd-4fcb-869c-d592-97336026cddf (at 6877@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f785f4400, cur 1457068846 expire 1457067946 last 1457067494 <4>[796216.864403] Lustre: atlas1-MDT0000: haven't heard from client 4258c6d3-a1e7-f3ae-7afa-b9d09b65373a (at 3006@gni100) in 1350 seconds. I think it's dead, and I am evicting it. exp ffff883f7abd2000, cur 1457069297 expire 1457068397 last 1457067947 <4>[796216.889594] Lustre: Skipped 128 previous similar messages <3>[811411.113777] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9298@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8833bd071d40/0xe00c22512a2d8a6d lrc: 4/0,0 mode: PR/PR res: [0x20014e592:0x2:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x60200000000020 nid: 9298@gni100 remote: 0xba0c01dc62df1b3d expref: 417521 pid: 16185 timeout: 5106077364 lvb_type: 0 <3>[811411.156897] LustreError: 15269:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9298@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8833bd071d40/0xe00c22512a2d8a6d lrc: 4/0,0 mode: PR/PR res: [0x20014e592:0x2:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x60200000000020 nid: 9298@gni100 remote: 0xba0c01dc62df1b3d expref: 417520 pid: 16185 timeout: 5106077364 lvb_type: 0 <3>[811411.198463] LustreError: 15269:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 1 previous similar message <4>[824055.576018] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[824060.648851] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[824062.804130] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[838441.761616] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[838445.479008] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[838447.523272] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[852841.466084] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[852845.180416] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[871767.796020] Lustre: atlas1-MDT0000: haven't heard from client 8d67aeb2-3dce-dce2-2836-f2a9655fbf22 (at 10.38.144.22@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f308a6800, cur 1457144848 expire 1457143948 last 1457143496 <4>[871767.821898] Lustre: Skipped 7 previous similar messages <4>[874150.379759] Lustre: atlas1-MDT0000: Client a34823e8-9831-e7bc-8cda-120d41dc234c (at 4306@gni100) reconnecting <4>[878246.657396] Lustre: atlas1-MDT0000: Client 21029be7-6edd-0e04-8932-365b1b23a215 (at 7448@gni100) reconnecting <4>[890417.776486] Lustre: atlas1-MDT0000: haven't heard from client d3c996e7-a7f0-f2f8-93e5-43fdcdc5efc9 (at 9307@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ee61c00, cur 1457163498 expire 1457162598 last 1457162146 <4>[896143.770091] Lustre: atlas1-MDT0000: haven't heard from client 31e1c1e7-b470-acca-bd7d-f0de5f012072 (at 16397@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7cd7e000, cur 1457169224 expire 1457168324 last 1457167872 <4>[907251.760551] Lustre: atlas1-MDT0000: haven't heard from client 8a2d6c11-2443-9e82-28b9-6dfedddccfd9 (at 16396@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e1a5000, cur 1457180332 expire 1457179432 last 1457178980 <4>[910447.068659] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[910452.094831] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[910454.269640] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[924842.655203] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[924846.428663] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[924848.546569] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[938703.993959] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 10791@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff881092a23b80/0xe00c2255e9f9ddf0 lrc: 4/0,0 mode: PR/PR res: [0x2002720fa:0x14fe3:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x60200000000020 nid: 10791@gni100 remote: 0x6f14fc9529dfaa56 expref: 76 pid: 15675 timeout: 5233370243 lvb_type: 0 <3>[938704.039826] LustreError: 15542:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 10791@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff881092a23b80/0xe00c2255e9f9ddf0 lrc: 4/0,0 mode: PR/PR res: [0x2002720fa:0x14fe3:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x60200000000020 nid: 10791@gni100 remote: 0x6f14fc9529dfaa56 expref: 74 pid: 15675 timeout: 5233370243 lvb_type: 0 <3>[938704.081583] LustreError: 15542:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 2 previous similar messages <4>[939241.892812] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[939245.699991] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[939247.865141] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[946919.724374] Lustre: atlas1-MDT0000: haven't heard from client 2c20f323-e54d-ca3f-3270-d96a2b216c24 (at 8160@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ccfb000, cur 1457220000 expire 1457219100 last 1457218648 <4>[950260.719883] Lustre: atlas1-MDT0000: haven't heard from client 1f190a7f-cd36-b5e4-936f-2930e5d47ee0 (at 9306@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ee5b400, cur 1457223341 expire 1457222441 last 1457221989 <4>[951967.719017] Lustre: atlas1-MDT0000: haven't heard from client 14d12ff6-1741-3be2-adb2-d57a5934ccb6 (at 9299@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ee54800, cur 1457225048 expire 1457224148 last 1457223696 <4>[962744.708496] Lustre: atlas1-MDT0000: haven't heard from client e97fd929-e316-9d6f-f3a2-8ea849414316 (at 10066@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e3b4800, cur 1457235825 expire 1457234925 last 1457234473 <4>[981955.825825] Lustre: atlas1-MDT0000: Client b5489f0c-878e-5028-fa9f-aed125a61ecf (at 12050@gni100) reconnecting <4>[996844.605751] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[996848.315393] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[996850.633932] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1005107.668841] Lustre: atlas1-MDT0000: haven't heard from client 2a111f86-ce2a-065c-b948-48ee8c0a19ef (at 5698@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a9b7400, cur 1457278188 expire 1457277288 last 1457276836 <4>[1011244.037044] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1011247.780849] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1011249.876297] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1020716.655536] Lustre: atlas1-MDT0000: haven't heard from client a10d2ea6-d798-d584-af23-f7054794262c (at 13194@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ae20000, cur 1457293797 expire 1457292897 last 1457292445 <4>[1023433.182240] Lustre: atlas1-MDT0000: Client 74b7233f-1871-02ea-fa7d-7afdd823b5e8 (at 10.36.205.206@o2ib) reconnecting <4>[1024939.918454] Lustre: atlas1-MDT0000: Client 094d79ca-cc18-ea41-8afb-d5aaf6beb9ae (at 7725@gni100) reconnecting <4>[1025642.782810] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1025646.559540] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1025648.689567] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1074706.608599] Lustre: atlas1-MDT0000: haven't heard from client 0350953b-dcf0-f06e-2675-99351cbf70ec (at 11796@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7cd5b000, cur 1457347787 expire 1457346887 last 1457346435 <6>[1080723.213738] Lustre: atlas1-MDT0000-mdc-ffff883f43708800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1080723.213829] Lustre: client wants to enable acl, but mdt not! <6>[1080723.213830] Lustre: Layout lock feature supported. <6>[1080723.237490] Lustre: Skipped 1008 previous similar messages <4>[1080723.351499] Lustre: Mounted atlas1-client <4>[1083243.467091] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1083247.136217] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1083249.147902] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1084054.459195] Lustre: Unmounted atlas1-client <4>[1097640.716820] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1097644.422423] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1097646.480481] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1108528.412725] Lustre: atlas1-MDT0000: Client c920a0f0-3628-9a29-68dd-ea59ed5d87ab (at 10.38.145.18@o2ib5) reconnecting <4>[1112044.012157] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1112047.704302] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1112049.714008] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1118087.254856] Lustre: atlas1-MDT0000: Client b23bff30-d25c-3ff7-347d-908c6005c877 (at 10.36.205.200@o2ib) reconnecting <4>[1119005.401496] Lustre: atlas1-MDT0000: Client b23bff30-d25c-3ff7-347d-908c6005c877 (at 10.36.205.200@o2ib) reconnecting <4>[1129272.841800] Lustre: atlas1-MDT0000: Client fc5829b1-d4f3-a976-168c-adc2a03d4cef (at 10.36.202.173@o2ib) reconnecting <4>[1130010.174734] Lustre: atlas1-MDT0000: Client fc5829b1-d4f3-a976-168c-adc2a03d4cef (at 10.36.202.173@o2ib) reconnecting <4>[1130619.780717] Lustre: atlas1-MDT0000: Client fc5829b1-d4f3-a976-168c-adc2a03d4cef (at 10.36.202.173@o2ib) reconnecting <3>[1135212.446744] LustreError: 138-a: atlas1-MDT0000: A client on nid 9310@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[1169652.889971] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1169657.483154] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1169659.836675] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1174023.738516] Lustre: atlas1-MDT0000: Client 34921102-f2d7-1cce-4a04-83fd8a3efaa4 (at 16283@gni100) reconnecting <4>[1174809.519750] Lustre: atlas1-MDT0000: haven't heard from client 689da961-faaf-64a2-c80e-ce250bc4562e (at 9053@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f79f22c00, cur 1457447890 expire 1457446990 last 1457446538 <4>[1174809.545076] Lustre: Skipped 1 previous similar message <4>[1176096.887449] Lustre: atlas1-MDT0000: Client d950a985-4c21-c664-35ea-5532b3251346 (at 10.36.205.198@o2ib) reconnecting <4>[1176449.516889] Lustre: atlas1-MDT0000: haven't heard from client a025e966-1e53-d13f-44c8-a3d382a1afc6 (at 290@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f486e9800, cur 1457449530 expire 1457448630 last 1457448178 <4>[1176449.541882] Lustre: Skipped 22 previous similar messages <3>[1176541.018284] LustreError: 138-a: atlas1-MDT0000: A client on nid 10.38.145.233@o2ib5 was evicted due to a lock blocking callback time out: rc -107 <3>[1176541.033568] LustreError: Skipped 1 previous similar message <4>[1179535.513867] Lustre: atlas1-MDT0000: haven't heard from client eeb6ecb3-f27e-da79-d17f-1ef7e0b8c484 (at 7731@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f798e3800, cur 1457452616 expire 1457451716 last 1457451264 <4>[1179535.539237] Lustre: Skipped 15 previous similar messages <4>[1182370.167724] Lustre: atlas1-MDT0000: Client e34d7cdb-494d-a4ad-549d-2135beffde4f (at 17046@gni100) reconnecting <4>[1184048.054262] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1184051.713539] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[1185123.425650] Lustre: atlas1-MDT0000-mdc-ffff883f3655a000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1185123.425797] Lustre: client wants to enable acl, but mdt not! <6>[1185123.425799] Lustre: Layout lock feature supported. <6>[1185123.449424] Lustre: Skipped 1008 previous similar messages <4>[1185123.566462] Lustre: Mounted atlas1-client <4>[1187373.458287] ptlrpcd_0: page allocation failure. order:1, mode:0x20 <4>[1187373.458326] ptlrpcd_5: page allocation failure. order:1, mode:0x20 <4>[1187373.458328] Pid: 14893, comm: ptlrpcd_5 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[1187373.458329] Call Trace: <4>[1187373.458335] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[1187373.458344] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[1187373.458348] [] ? kmem_getpages+0x62/0x170 <4>[1187373.458349] [] ? fallback_alloc+0x1ba/0x270 <4>[1187373.458350] [] ? cache_grow+0x2cf/0x320 <4>[1187373.458352] [] ? ____cache_alloc_node+0x99/0x160 <4>[1187373.458354] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[1187373.458355] [] ? __kmalloc+0x199/0x230 <4>[1187373.458357] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[1187373.458358] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[1187373.458390] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[1187373.458407] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[1187373.458424] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[1187373.458440] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[1187373.458442] ptlrpcd_6: page allocation failure. order:1, mode:0x20 <4>[1187373.458461] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[1187373.458462] Pid: 14894, comm: ptlrpcd_6 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[1187373.458464] Call Trace: <4>[1187373.458481] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[1187373.458485] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[1187373.458488] [] ? default_wake_function+0x0/0x20 <4>[1187373.458493] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[1187373.458510] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[1187373.458513] [] ? kmem_getpages+0x62/0x170 <4>[1187373.458516] [] ? kthread+0x9e/0xc0 <4>[1187373.458517] [] ? fallback_alloc+0x1ba/0x270 <4>[1187373.458519] [] ? child_rip+0xa/0x20 <4>[1187373.458521] [] ? cache_grow+0x2cf/0x320 <4>[1187373.458523] [] ? kthread+0x0/0xc0 <4>[1187373.458524] [] ? ____cache_alloc_node+0x99/0x160 <4>[1187373.458526] [] ? child_rip+0x0/0x20 <6>[1187373.458527] Mem-Info: <4>[1187373.458528] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[1187373.458529] Node 0 DMA per-cpu: <4>[1187373.458531] [] ? __kmalloc+0x199/0x230 <4>[1187373.458532] CPU 0: hi: 0, btch: 1 usd: 0 <4>[1187373.458533] CPU 1: hi: 0, btch: 1 usd: 0 <4>[1187373.458535] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[1187373.458536] CPU 2: hi: 0, btch: 1 usd: 0 <4>[1187373.458537] CPU 3: hi: 0, btch: 1 usd: 0 <4>[1187373.458538] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[1187373.458539] CPU 4: hi: 0, btch: 1 usd: 0 <4>[1187373.458540] CPU 5: hi: 0, btch: 1 usd: 0 <4>[1187373.458562] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[1187373.458563] CPU 6: hi: 0, btch: 1 usd: 0 <4>[1187373.458564] CPU 7: hi: 0, btch: 1 usd: 0 <4>[1187373.458581] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[1187373.458582] Node 0 DMA32 per-cpu: <4>[1187373.458598] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[1187373.458599] CPU 0: hi: 186, btch: 31 usd: 0 <4>[1187373.458601] CPU 1: hi: 186, btch: 31 usd: 0 <4>[1187373.458617] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[1187373.458617] CPU 2: hi: 186, btch: 31 usd: 0 <4>[1187373.458618] CPU 3: hi: 186, btch: 31 usd: 0 <4>[1187373.458636] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[1187373.458637] CPU 4: hi: 186, btch: 31 usd: 0 <4>[1187373.458638] CPU 5: hi: 186, btch: 31 usd: 0 <4>[1187373.458656] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[1187373.458657] CPU 6: hi: 186, btch: 31 usd: 0 <4>[1187373.458658] CPU 7: hi: 186, btch: 31 usd: 0 <4>[1187373.458660] [] ? default_wake_function+0x0/0x20 <4>[1187373.458661] Node 0 Normal per-cpu: <4>[1187373.458679] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[1187373.458680] CPU 0: hi: 186, btch: 31 usd: 35 <4>[1187373.458681] CPU 1: hi: 186, btch: 31 usd: 86 <4>[1187373.458683] [] ? kthread+0x9e/0xc0 <4>[1187373.458684] CPU 2: hi: 186, btch: 31 usd: 144 <4>[1187373.458685] CPU 3: hi: 186, btch: 31 usd: 96 <4>[1187373.458686] [] ? child_rip+0xa/0x20 <4>[1187373.458687] CPU 4: hi: 186, btch: 31 usd: 147 <4>[1187373.458688] CPU 5: hi: 186, btch: 31 usd: 28 <4>[1187373.458690] [] ? kthread+0x0/0xc0 <4>[1187373.458690] CPU 6: hi: 186, btch: 31 usd: 63 <4>[1187373.458692] CPU 7: hi: 186, btch: 31 usd: 157 <4>[1187373.458693] [] ? child_rip+0x0/0x20 <4>[1187373.458695] active_anon:4975665 inactive_anon:91731 isolated_anon:0 <4>[1187373.458695] active_file:10037800 inactive_file:30950042 isolated_file:0 <4>[1187373.458696] unevictable:10819 dirty:8779 writeback:725 unstable:0 <4>[1187373.458696] free:466899 slab_reclaimable:5186045 slab_unreclaimable:13491890 <4>[1187373.458697] mapped:8917 shmem:160796 pagetables:10917 bounce:0 <6>[1187373.458697] Mem-Info: <4>[1187373.458698] Node 0 Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1187373.458702] DMA per-cpu: <4>[1187373.458702] lowmem_reserve[]:CPU 0: hi: 0, btch: 1 usd: 0 <4>[1187373.458704] 0CPU 1: hi: 0, btch: 1 usd: 0 <4>[1187373.458705] 1880CPU 2: hi: 0, btch: 1 usd: 0 <4>[1187373.458706] 258420CPU 3: hi: 0, btch: 1 usd: 0 <4>[1187373.458707] 258420CPU 4: hi: 0, btch: 1 usd: 0 <4>[1187373.458708] <4>[1187373.458709] CPU 5: hi: 0, btch: 1 usd: 0 <4>[1187373.458710] Node 0 CPU 6: hi: 0, btch: 1 usd: 0 <4>[1187373.458712] DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1187373.458714] CPU 7: hi: 0, btch: 1 usd: 0 <4>[1187373.458715] lowmem_reserve[]:Node 0 0DMA32 per-cpu: <4>[1187373.458716] 0CPU 0: hi: 186, btch: 31 usd: 0 <4>[1187373.458717] 256540CPU 1: hi: 186, btch: 31 usd: 0 <4>[1187373.458719] 256540CPU 2: hi: 186, btch: 31 usd: 0 <4>[1187373.458720] <4>[1187373.458720] CPU 3: hi: 186, btch: 31 usd: 0 <4>[1187373.458721] Node 0 CPU 4: hi: 186, btch: 31 usd: 0 <4>[1187373.458724] Normal free:1466740kB min:67088kB low:83860kB high:100632kB active_anon:19902660kB inactive_anon:366924kB active_file:40151200kB inactive_file:123800168kB unevictable:43276kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:35116kB writeback:2900kB mapped:35668kB shmem:643184kB slab_reclaimable:20744180kB slab_unreclaimable:53967560kB kernel_stack:74144kB pagetables:43668kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no <4>[1187373.458726] CPU 5: hi: 186, btch: 31 usd: 0 <4>[1187373.458727] lowmem_reserve[]:CPU 6: hi: 186, btch: 31 usd: 0 <4>[1187373.458728] 0CPU 7: hi: 186, btch: 31 usd: 0 <4>[1187373.458729] 0Node 0 0Normal per-cpu: <4>[1187373.458731] 0CPU 0: hi: 186, btch: 31 usd: 50 <4>[1187373.458732] <4>[1187373.458732] CPU 1: hi: 186, btch: 31 usd: 86 <4>[1187373.458733] Node 0 CPU 2: hi: 186, btch: 31 usd: 144 <4>[1187373.458734] DMA: CPU 3: hi: 186, btch: 31 usd: 96 <4>[1187373.458736] 1*4kB CPU 4: hi: 186, btch: 31 usd: 147 <4>[1187373.458737] 1*8kB CPU 5: hi: 186, btch: 31 usd: 48 <4>[1187373.458738] 1*16kB CPU 6: hi: 186, btch: 31 usd: 63 <4>[1187373.458739] 1*32kB CPU 7: hi: 186, btch: 31 usd: 175 <4>[1187373.458740] 1*64kB active_anon:4975665 inactive_anon:91731 isolated_anon:0 <4>[1187373.458742] active_file:10037800 inactive_file:30950042 isolated_file:0 <4>[1187373.458743] unevictable:10819 dirty:8779 writeback:725 unstable:0 <4>[1187373.458743] free:466899 slab_reclaimable:5186045 slab_unreclaimable:13491890 <4>[1187373.458744] mapped:8917 shmem:160796 pagetables:10917 bounce:0 <4>[1187373.458744] 0*128kB Node 0 1*256kB DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1187373.458749] 0*512kB lowmem_reserve[]:1*1024kB 01*2048kB 18803*4096kB 258420= 15740kB <4>[1187373.458752] 258420Node 0 <4>[1187373.458753] DMA32: Node 0 9*4kB DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1187373.458758] 3*8kB lowmem_reserve[]:4*16kB 07*32kB 06*64kB 2565405*128kB 2565407*256kB <4>[1187373.458762] 10*512kB Node 0 8*1024kB Normal free:1466740kB min:67088kB low:83860kB high:100632kB active_anon:19902660kB inactive_anon:366924kB active_file:40151200kB inactive_file:123800168kB unevictable:43276kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:35116kB writeback:2900kB mapped:35668kB shmem:643184kB slab_reclaimable:20744180kB slab_unreclaimable:53967560kB kernel_stack:74144kB pagetables:43668kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no <4>[1187373.458767] 4*2048kB lowmem_reserve[]:88*4096kB 0= 385116kB <4>[1187373.458769] 0Node 0 0Normal: 0366546*4kB <4>[1187373.458771] 0*8kB Node 0 0*16kB DMA: 0*32kB 1*4kB 0*64kB 1*8kB 0*128kB 1*16kB 0*256kB 1*32kB 0*512kB 1*64kB 0*1024kB 0*128kB 0*2048kB 1*256kB 0*4096kB 0*512kB = 1466184kB <4>[1187373.458779] 1*1024kB 41149757 total pagecache pages <4>[1187373.458780] 1*2048kB 0 pages in swap cache <4>[1187373.458781] 3*4096kB Swap cache stats: add 0, delete 0, find 0/0 <4>[1187373.458782] = 15740kB <4>[1187373.458783] Free swap = 0kB <4>[1187373.458784] Node 0 Total swap = 0kB <4>[1187373.458784] DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[1187373.458787] Node 0 Normal: 366569*4kB 0*8kB 0*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1466308kB <4>[1187373.458790] 41149757 total pagecache pages <4>[1187373.458791] 0 pages in swap cache <4>[1187373.458791] Swap cache stats: add 0, delete 0, find 0/0 <4>[1187373.458792] Free swap = 0kB <4>[1187373.458792] Total swap = 0kB <6>[1187373.796302] 67108863 pages RAM <6>[1187373.796303] 1015653 pages reserved <6>[1187373.796304] 40474981 pages shared <6>[1187373.796304] 24913281 pages non-shared <3>[1187373.796324] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[1187373.796326] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1376454232 total bytes allocated by lnet <6>[1187374.132265] 67108863 pages RAM <6>[1187374.132267] 1015653 pages reserved <6>[1187374.132267] 40474651 pages shared <6>[1187374.132268] 24917678 pages non-shared <4>[1187374.716368] Pid: 14888, comm: ptlrpcd_0 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[1187374.725945] Call Trace: <4>[1187374.729094] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[1187374.736810] [] ? __alloc_pages_nodemask+0x129/0x950 <4>[1187374.744517] [] ? kmem_getpages+0x62/0x170 <4>[1187374.751258] [] ? fallback_alloc+0x1ba/0x270 <4>[1187374.758189] [] ? cache_grow+0x2cf/0x320 <4>[1187374.764735] [] ? ____cache_alloc_node+0x99/0x160 <4>[1187374.772159] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[1187374.780771] [] ? __kmalloc+0x199/0x230 <4>[1187374.787226] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[1187374.795835] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[1187374.803975] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[1187374.811613] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[1187374.819723] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[1187374.828255] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[1187374.836388] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[1187374.844126] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[1187374.851308] [] ? default_wake_function+0x0/0x20 <4>[1187374.858655] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[1187374.865592] [] ? kthread+0x9e/0xc0 <4>[1187374.871656] [] ? child_rip+0xa/0x20 <4>[1187374.877815] [] ? kthread+0x0/0xc0 <4>[1187374.883782] [] ? child_rip+0x0/0x20 <6>[1187374.889944] Mem-Info: <4>[1187374.892895] Node 0 DMA per-cpu: <4>[1187374.896842] CPU 0: hi: 0, btch: 1 usd: 0 <4>[1187374.902614] CPU 1: hi: 0, btch: 1 usd: 0 <4>[1187374.908383] CPU 2: hi: 0, btch: 1 usd: 0 <4>[1187374.914153] CPU 3: hi: 0, btch: 1 usd: 0 <4>[1187374.919923] CPU 4: hi: 0, btch: 1 usd: 0 <4>[1187374.925693] CPU 5: hi: 0, btch: 1 usd: 0 <4>[1187374.931464] CPU 6: hi: 0, btch: 1 usd: 0 <4>[1187374.937235] CPU 7: hi: 0, btch: 1 usd: 0 <4>[1187374.943002] Node 0 DMA32 per-cpu: <4>[1187374.947135] CPU 0: hi: 186, btch: 31 usd: 0 <4>[1187374.952903] CPU 1: hi: 186, btch: 31 usd: 0 <4>[1187374.958676] CPU 2: hi: 186, btch: 31 usd: 0 <4>[1187374.964444] CPU 3: hi: 186, btch: 31 usd: 0 <4>[1187374.970214] CPU 4: hi: 186, btch: 31 usd: 0 <4>[1187374.981223] CPU 5: hi: 186, btch: 31 usd: 0 <4>[1187374.986997] CPU 6: hi: 186, btch: 31 usd: 0 <4>[1187374.992763] CPU 7: hi: 186, btch: 31 usd: 0 <4>[1187374.998532] Node 0 Normal per-cpu: <4>[1187375.002758] CPU 0: hi: 186, btch: 31 usd: 168 <4>[1187375.008528] CPU 1: hi: 186, btch: 31 usd: 125 <4>[1187375.014298] CPU 2: hi: 186, btch: 31 usd: 185 <4>[1187375.020068] CPU 3: hi: 186, btch: 31 usd: 96 <4>[1187375.025838] CPU 4: hi: 186, btch: 31 usd: 4 <4>[1187375.031607] CPU 5: hi: 186, btch: 31 usd: 45 <4>[1187375.037380] CPU 6: hi: 186, btch: 31 usd: 2 <4>[1187375.043156] CPU 7: hi: 186, btch: 31 usd: 157 <4>[1187375.048924] active_anon:4975665 inactive_anon:91731 isolated_anon:0 <4>[1187375.048925] active_file:10037863 inactive_file:30950250 isolated_file:0 <4>[1187375.048925] unevictable:10819 dirty:10829 writeback:2791 unstable:0 <4>[1187375.048925] free:465983 slab_reclaimable:5185625 slab_unreclaimable:13493019 <4>[1187375.048926] mapped:8917 shmem:160796 pagetables:10917 bounce:0 <4>[1187375.087502] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1187375.129546] lowmem_reserve[]: 0 1880 258420 258420 <4>[1187375.135368] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[1187375.178493] lowmem_reserve[]: 0 0 256540 256540 <4>[1187375.184032] Node 0 Normal free:1457620kB min:67088kB low:83860kB high:100632kB active_anon:19902660kB inactive_anon:366924kB active_file:40151452kB inactive_file:123805268kB unevictable:43276kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:47376kB writeback:11164kB mapped:35668kB shmem:643184kB slab_reclaimable:20742500kB slab_unreclaimable:53973240kB kernel_stack:74176kB pagetables:43668kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[1187375.235126] lowmem_reserve[]: 0 0 0 0 <4>[1187375.239701] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[1187375.252353] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[1187375.265490] Node 0 Normal: 355713*4kB 9*8kB 1*16kB 1*32kB 365*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1446332kB <4>[1187375.279312] 41153067 total pagecache pages <4>[1187375.284309] 0 pages in swap cache <4>[1187375.288427] Swap cache stats: add 0, delete 0, find 0/0 <4>[1187375.294686] Free swap = 0kB <4>[1187375.298322] Total swap = 0kB <6>[1187375.637214] 67108863 pages RAM <6>[1187375.641043] 1015653 pages reserved <6>[1187375.645256] 40475938 pages shared <6>[1187375.649373] 24918323 pages non-shared <3>[1187375.653921] LustreError: 14888:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[1187375.678677] LustreError: 14888:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <3>[1187375.690753] LustreError: 14888:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1376361912 total bytes allocated by lnet <3>[1187375.703754] LustreError: 14888:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <4>[1188161.507114] Lustre: atlas1-MDT0000: haven't heard from client ef155913-86b1-29da-c6ab-91bab3e12a46 (at 10.38.145.17@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3a3e0000, cur 1457461242 expire 1457460342 last 1457459890 <4>[1188161.533075] Lustre: Skipped 11 previous similar messages <4>[1188487.390077] Lustre: Unmounted atlas1-client <4>[1190470.750474] Lustre: atlas1-MDT0000: Client a8d61059-5277-e781-48cb-a31e4995ebcc (at 9306@gni100) reconnecting <4>[1198444.153417] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1198447.794765] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1200525.499056] Lustre: atlas1-MDT0000: haven't heard from client afed78fb-c2d0-5189-58b3-7e6358cb24bb (at 19073@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3b14ec00, cur 1457473606 expire 1457472706 last 1457472254 <3>[1200779.758238] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9311@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff881d382fbb40/0xe00c225d4acd22a7 lrc: 4/0,0 mode: PR/PR res: [0x2002b0f4c:0x15de9:0x0].0 bits 0x13 rrc: 15 type: IBT flags: 0x60200000000020 nid: 9311@gni100 remote: 0x441f37e83731ee57 expref: 246229 pid: 16162 timeout: 5495446509 lvb_type: 0 <3>[1200779.801931] LustreError: 15669:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9311@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff881d382fbb40/0xe00c225d4acd22a7 lrc: 4/0,0 mode: PR/PR res: [0x2002b0f4c:0x15de9:0x0].0 bits 0x13 rrc: 15 type: IBT flags: 0x60200000000020 nid: 9311@gni100 remote: 0x441f37e83731ee57 expref: 246166 pid: 16162 timeout: 5495446509 lvb_type: 0 <4>[1256054.270872] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1256058.369526] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1256060.760250] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1270443.303838] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1270447.227651] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1270449.309497] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1275870.450186] Lustre: atlas1-MDT0000: Client 074e9b25-3977-e81d-1faf-e6440af7b783 (at 10.38.145.47@o2ib5) reconnecting <4>[1278401.420173] Lustre: atlas1-MDT0000: haven't heard from client 6db14ba2-a3f6-8d19-b29c-67e11b51eb1f (at 13939@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c339c00, cur 1457551482 expire 1457550582 last 1457550130 <4>[1284031.414507] Lustre: atlas1-MDT0000: haven't heard from client bb88676a-1a21-802a-e7c8-f2af81220df0 (at 3556@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a096800, cur 1457557112 expire 1457556212 last 1457555760 <4>[1284854.904911] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1284858.910267] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1284861.046752] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[1285921.924955] LustreError: 138-a: atlas1-MDT0000: A client on nid 10.38.145.17@o2ib5 was evicted due to a lock blocking callback time out: rc -107 <6>[1285923.048546] Lustre: atlas1-MDT0000-mdc-ffff8836d20df400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1285923.048799] Lustre: client wants to enable acl, but mdt not! <6>[1285923.048803] Lustre: Layout lock feature supported. <6>[1285923.072303] Lustre: Skipped 1008 previous similar messages <4>[1285923.187603] Lustre: Mounted atlas1-client <4>[1289326.474313] Lustre: Unmounted atlas1-client <4>[1289856.034321] Lustre: atlas1-MDT0000: Client b23bff30-d25c-3ff7-347d-908c6005c877 (at 10.36.205.200@o2ib) reconnecting <4>[1297961.416772] Lustre: atlas1-MDT0000: Client b23bff30-d25c-3ff7-347d-908c6005c877 (at 10.36.205.200@o2ib) reconnecting <4>[1331017.369659] Lustre: atlas1-MDT0000: haven't heard from client 27dde1a2-7126-9483-18a3-6b5b8f879bf5 (at 9556@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f78c00800, cur 1457604098 expire 1457603198 last 1457602746 <4>[1331017.394940] Lustre: Skipped 3 previous similar messages <4>[1337574.835829] Lustre: atlas1-MDT0000: Client 40093f82-98ee-ecdb-f508-a200ec6e67cb (at 5650@gni100) reconnecting <4>[1342449.549482] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1342453.129673] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1356853.078038] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1356856.721247] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1359782.341460] Lustre: atlas1-MDT0000: haven't heard from client 0558c164-3552-1ada-041d-1fa503540b51 (at 10.38.144.21@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f342ad800, cur 1457632863 expire 1457631963 last 1457631511 <4>[1363083.338319] Lustre: atlas1-MDT0000: haven't heard from client 50f03228-1c6d-da2e-c5ed-e1ff20e1ab6a (at 10.38.144.22@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3fafb000, cur 1457636164 expire 1457635264 last 1457634812 <4>[1363083.364359] Lustre: Skipped 1 previous similar message <4>[1371013.329183] Lustre: atlas1-MDT0000: haven't heard from client 2b06605f-87a7-77c8-1a87-d1336ea0cd2e (at 10.38.145.233@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f312dc800, cur 1457644094 expire 1457643194 last 1457642742 <4>[1371240.811924] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1371244.399616] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1383996.319009] Lustre: atlas1-MDT0000: haven't heard from client 9397b537-e9a6-0ff3-4dcf-209101a48cd8 (at 12991@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7bfc2400, cur 1457657077 expire 1457656177 last 1457655725 <4>[1388139.312740] Lustre: atlas1-MDT0000: haven't heard from client 07f98da5-5d4a-9924-5dbc-dfd5f963169b (at 13876@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a9c3400, cur 1457661220 expire 1457660320 last 1457659868 <4>[1391365.309064] Lustre: atlas1-MDT0000: haven't heard from client 4e6244ae-72ed-9757-123d-e8eb5f19f9bb (at 16206@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7eb82400, cur 1457664446 expire 1457663546 last 1457663094 <4>[1403579.296781] Lustre: atlas1-MDT0000: haven't heard from client cf23d46f-1fa8-771f-9036-855f7b33500a (at 14226@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c6c6c00, cur 1457676660 expire 1457675760 last 1457675308 <4>[1414595.285312] Lustre: atlas1-MDT0000: haven't heard from client 338a58bd-db17-dddb-2e03-6c8af98d8839 (at 4888@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7b8dd400, cur 1457687676 expire 1457686776 last 1457686324 <6>[1426322.223395] Lustre: atlas1-MDT0000-mdc-ffff883f3bebfc00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1426322.223531] Lustre: client wants to enable acl, but mdt not! <6>[1426322.223533] Lustre: Layout lock feature supported. <6>[1426322.247161] Lustre: Skipped 1008 previous similar messages <4>[1426322.364491] Lustre: Mounted atlas1-client <4>[1428887.062902] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1428890.782204] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1428893.213826] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1430275.855879] Lustre: Unmounted atlas1-client <4>[1440559.160562] Lustre: atlas1-MDT0000: Client 150e4e6d-ec0f-7c39-e75d-68ef0aefeb2f (at 7851@gni100) reconnecting <4>[1443260.476032] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1443267.956747] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1443270.696568] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1452050.244295] Lustre: atlas1-MDT0000: haven't heard from client dbbf00de-32ee-1006-cef3-90b9412496bb (at 31@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883feba50800, cur 1457725131 expire 1457724231 last 1457723779 <4>[1457651.244783] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1457656.187671] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1457659.239100] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[1461324.496731] LustreError: 18919:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 83@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff88136cb66080/0xe00c2267defab18f lrc: 4/0,0 mode: PR/PR res: [0x2002bd28f:0x1df64:0x0].0 bits 0x1b rrc: 4 type: IBT flags: 0x60200000000020 nid: 83@gni100 remote: 0x98e20867632fca52 expref: 503950 pid: 15377 timeout: 5755991368 lvb_type: 0 <3>[1461324.540276] LustreError: 15610:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 83@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff88136cb66080/0xe00c2267defab18f lrc: 4/0,0 mode: PR/PR res: [0x2002bd28f:0x1df64:0x0].0 bits 0x1b rrc: 4 type: IBT flags: 0x60200000000020 nid: 83@gni100 remote: 0x98e20867632fca52 expref: 503947 pid: 15377 timeout: 5755991368 lvb_type: 0 <4>[1463247.231764] Lustre: atlas1-MDT0000: haven't heard from client a8d61059-5277-e781-48cb-a31e4995ebcc (at 9306@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3d313000, cur 1457736328 expire 1457735428 last 1457734976 <3>[1463468.493307] LustreError: 15627:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9299@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff880471762500/0xe00c22668d54a098 lrc: 4/0,0 mode: PR/PR res: [0x2002c8cf4:0x1794:0x0].0 bits 0x13 rrc: 8 type: IBT flags: 0x60200000000020 nid: 9299@gni100 remote: 0x9f3c6fb9b3c8a6f6 expref: 196520 pid: 15329 timeout: 5758135534 lvb_type: 0 <3>[1463468.537391] LustreError: 15696:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9299@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff880471762500/0xe00c22668d54a098 lrc: 4/0,0 mode: PR/PR res: [0x2002c8cf4:0x1794:0x0].0 bits 0x13 rrc: 8 type: IBT flags: 0x60200000000020 nid: 9299@gni100 remote: 0x9f3c6fb9b3c8a6f6 expref: 196268 pid: 15329 timeout: 5758135534 lvb_type: 0 <4>[1465137.229497] Lustre: atlas1-MDT0000: haven't heard from client 3e73befd-6006-a111-0d5e-58345cfa3f52 (at 95@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ee5b000, cur 1457738218 expire 1457737318 last 1457736866 <4>[1502758.191764] Lustre: atlas1-MDT0000: haven't heard from client 1dea046b-64c9-f42c-b55b-13e91bc81d14 (at 9450@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7dcab800, cur 1457775839 expire 1457774939 last 1457774487 <4>[1513326.897703] Lustre: atlas1-MDT0000: Client c70ea121-bb3b-66db-8d3d-ee9eef7a7952 (at 12247@gni100) reconnecting <4>[1515263.874077] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1515268.522779] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1515270.633726] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1529641.235971] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1529645.123865] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1529647.213166] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1544045.488965] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1544049.910760] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1544052.053896] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1553749.210660] Lustre: atlas1-MDT0000: Client 61fa7e54-a7f0-1753-0e61-441b850c495f (at 4497@gni100) reconnecting <4>[1565519.125096] Lustre: atlas1-MDT0000: haven't heard from client 8a89499c-4be0-ef40-1f0a-e199b6ea1028 (at 2763@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d3dd400, cur 1457838600 expire 1457837700 last 1457837248 <4>[1589482.101691] Lustre: atlas1-MDT0000: haven't heard from client 8b8e4cdc-61d4-3903-1b6e-083e351871af (at 15209@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f786d8c00, cur 1457862563 expire 1457861663 last 1457861211 <4>[1589933.100775] Lustre: atlas1-MDT0000: haven't heard from client d8008724-9501-4741-c327-77e95a64dba0 (at 8378@gni100) in 1349 seconds. I think it's dead, and I am evicting it. exp ffff883f7e7af400, cur 1457863014 expire 1457862114 last 1457861665 <4>[1589933.126083] Lustre: Skipped 140 previous similar messages <6>[1595522.466602] Lustre: atlas1-MDT0000-mdc-ffff883fa79c3400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1595522.466718] Lustre: client wants to enable acl, but mdt not! <6>[1595522.466720] Lustre: Layout lock feature supported. <6>[1595522.490350] Lustre: Skipped 1008 previous similar messages <4>[1595522.605421] Lustre: Mounted atlas1-client <4>[1598048.461850] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1598052.461295] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1598054.806365] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1598567.163350] Lustre: Unmounted atlas1-client <4>[1601301.884230] Lustre: atlas1-MDT0000: Client f42d8eb3-ec69-7f01-479f-4209c720239c (at 13782@gni100) reconnecting <4>[1611238.079598] Lustre: atlas1-MDT0000: haven't heard from client 58968ca5-5f07-949f-98d4-bbeb4ad95a18 (at 7564@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d93a000, cur 1457884319 expire 1457883419 last 1457882967 <4>[1611238.105003] Lustre: Skipped 53 previous similar messages <4>[1612220.010922] Lustre: atlas1-MDT0000: Client 9460b433-55d1-5a6a-473e-8e288af32171 (at 10694@gni100) reconnecting <4>[1612450.742617] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1612454.371606] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1626847.255893] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1626850.884917] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1629764.059417] Lustre: atlas1-MDT0000: haven't heard from client 40286d3c-e78e-c7ff-a10b-aec86f97fa11 (at 14665@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7cca2c00, cur 1457902845 expire 1457901945 last 1457901493 <4>[1639155.898074] Lustre: atlas1-MDT0000: Client 094eaa3c-1810-bdc8-a53c-6007f5060803 (at 11529@gni100) reconnecting <4>[1643301.221610] Lustre: atlas1-MDT0000: Client e924493f-5f7e-2648-d1cd-a7d98248986e (at 8600@gni100) reconnecting <4>[1666222.020912] Lustre: atlas1-MDT0000: haven't heard from client d884341f-6a2e-9fa8-1499-c21da9714f7d (at 10069@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c1b2000, cur 1457939303 expire 1457938403 last 1457937951 <4>[1682558.700684] Lustre: atlas1-MDT0000: Client f071a835-7ec4-4bd6-c715-797dcefad6cb (at 16063@gni100) reconnecting <4>[1683407.232049] Lustre: atlas1-MDT0000: Client 2a4fda27-71d8-a7a7-2659-75121837df26 (at 9318@gni100) reconnecting <4>[1684219.018176] Lustre: atlas1-MDT0000: Client 5e96ed0c-4208-0af1-ccc7-e6f97a9b27ff (at 16703@gni100) reconnecting <4>[1684220.471043] Lustre: atlas1-MDT0000: Client 3bc94919-6f53-3096-c661-180e0b4876f3 (at 16652@gni100) reconnecting <4>[1684220.482887] Lustre: Skipped 1 previous similar message <4>[1684222.510902] Lustre: atlas1-MDT0000: Client a350842c-1017-e38c-8da0-bc7e6eff1248 (at 16688@gni100) reconnecting <4>[1684231.583058] Lustre: atlas1-MDT0000: Client 396a2fe4-3e8e-e0f1-4615-01bc7c95047d (at 16697@gni100) reconnecting <4>[1684238.186037] Lustre: atlas1-MDT0000: Client af70df13-752b-a088-12a4-c2447b02e9c0 (at 13541@gni100) reconnecting <4>[1684270.204363] Lustre: atlas1-MDT0000: Client 16e4fe1b-d938-2b24-db6b-75af1f0016d9 (at 15642@gni100) reconnecting <4>[1684448.617047] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1684452.175897] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1684637.325544] Lustre: atlas1-MDT0000: Client 26dc5e74-1ee6-eac1-57bc-64f47aff4b76 (at 15556@gni100) reconnecting <4>[1684637.337375] Lustre: Skipped 4 previous similar messages <4>[1684683.913125] Lustre: atlas1-MDT0000: Client 4f41a9c7-726f-bbf5-de20-b25c672e9c82 (at 10948@gni100) reconnecting <4>[1684683.924950] Lustre: Skipped 4 previous similar messages <4>[1684782.208415] Lustre: atlas1-MDT0000: Client b6902dce-ada4-afcf-3d38-af3b533a3ff1 (at 18149@gni100) reconnecting <4>[1684782.220241] Lustre: Skipped 2 previous similar messages <4>[1685400.004896] Lustre: atlas1-MDT0000: haven't heard from client e865c377-3ac8-989e-7686-172b4435c5ac (at 16698@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e2d8400, cur 1457958481 expire 1457957581 last 1457957129 <4>[1698841.671011] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1698845.196714] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1699603.987638] Lustre: atlas1-MDT0000: haven't heard from client 0b9e2f92-28c2-acb4-bb18-283f765037cf (at 10.36.207.152@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3307fc00, cur 1457972685 expire 1457971785 last 1457971333 <4>[1699604.013643] Lustre: Skipped 7 previous similar messages <6>[1699922.363538] Lustre: atlas1-MDT0000-mdc-ffff883f48d30400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1699922.363715] Lustre: client wants to enable acl, but mdt not! <6>[1699922.363717] Lustre: Layout lock feature supported. <6>[1699922.387336] Lustre: Skipped 1008 previous similar messages <4>[1699922.502703] Lustre: Mounted atlas1-client <4>[1703047.890206] Lustre: Unmounted atlas1-client <4>[1713249.742254] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1713253.813954] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1713255.988345] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1741721.944703] Lustre: atlas1-MDT0000: haven't heard from client 4652f469-9eb2-6a07-f3e2-8f1295f4b99e (at 9086@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e0cec00, cur 1458014803 expire 1458013903 last 1458013451 <6>[1745411.294151] sd 10:0:1:6: rdac: array Atlas1-mdt1, ctlr 0, queueing MODE_SELECT command <6>[1745411.303755] sd 10:0:1:6: rdac: array Atlas1-mdt1, ctlr 0, MODE_SELECT returned with sense 06/94/01 <6>[1745411.314438] sd 10:0:1:6: rdac: array Atlas1-mdt1, ctlr 0, retrying MODE_SELECT command <6>[1745411.558095] sd 10:0:1:6: rdac: array Atlas1-mdt1, ctlr 0, MODE_SELECT completed <6>[1765974.823980] sd 10:0:1:2: rdac: array Atlas1-mdt1, ctlr 0, queueing MODE_SELECT command <6>[1765974.833592] sd 10:0:1:2: rdac: array Atlas1-mdt1, ctlr 0, MODE_SELECT returned with sense 06/94/01 <6>[1765974.844278] sd 10:0:1:2: rdac: array Atlas1-mdt1, ctlr 0, retrying MODE_SELECT command <6>[1765975.084645] sd 10:0:1:2: rdac: array Atlas1-mdt1, ctlr 0, MODE_SELECT completed <4>[1766173.919313] Lustre: atlas1-MDT0000: haven't heard from client 1ba85ed4-e5b8-d516-db06-867bd7b1fb58 (at 5372@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c203000, cur 1458039255 expire 1458038355 last 1458037903 <4>[1770848.785719] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1770852.751934] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1778136.907155] Lustre: atlas1-MDT0000: haven't heard from client 5ced6222-1285-e3ae-72fa-a100b2a5cc70 (at 12926@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3f664400, cur 1458051218 expire 1458050318 last 1458049866 <4>[1778587.906583] Lustre: atlas1-MDT0000: haven't heard from client 7848be60-ee31-6716-6b33-0491eb8e1251 (at 204@gni3) in 1247 seconds. I think it's dead, and I am evicting it. exp ffff883f37aa4800, cur 1458051669 expire 1458050769 last 1458050422 <4>[1778587.931579] Lustre: Skipped 41 previous similar messages <4>[1780482.728931] Lustre: atlas1-MDT0000: Client ec405fec-3141-5302-3814-5a0272491e33 (at 2242@gni100) reconnecting <4>[1780482.740666] Lustre: Skipped 1 previous similar message <4>[1780781.831069] Lustre: atlas1-MDT0000: Client ec405fec-3141-5302-3814-5a0272491e33 (at 2242@gni100) reconnecting <4>[1780781.842863] Lustre: Skipped 33 previous similar messages <4>[1780973.760327] Lustre: atlas1-MDT0000: Client 4e4be631-e865-bd39-5adb-101725d04c16 (at 4939@gni100) reconnecting <4>[1785066.504341] Lustre: atlas1-MDT0000: Client 44220b03-dcea-77ab-a602-a0189fe92270 (at 11035@gni100) reconnecting <4>[1785195.255303] Lustre: atlas1-MDT0000: Client 3bc94919-6f53-3096-c661-180e0b4876f3 (at 16652@gni100) reconnecting <4>[1785195.267149] Lustre: Skipped 1 previous similar message <4>[1785211.333060] Lustre: atlas1-MDT0000: Client aa858c59-dab2-149d-ab12-5a127543acd6 (at 10468@gni100) reconnecting <4>[1785240.490933] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1785244.124985] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1785244.442854] Lustre: atlas1-MDT0000: Client 42661eb6-9a85-cd3e-9948-9e01e48b47d7 (at 16683@gni100) reconnecting <4>[1785244.454681] Lustre: Skipped 3 previous similar messages <4>[1785310.319885] Lustre: atlas1-MDT0000: Client e2986c6d-7efc-5531-bddd-f233d9a1a57b (at 16643@gni100) reconnecting <4>[1785310.331708] Lustre: Skipped 21 previous similar messages <4>[1785455.235111] Lustre: atlas1-MDT0000: Client 4312e208-56fc-597e-d0b7-461cd2fc554b (at 18714@gni100) reconnecting <4>[1785455.246956] Lustre: Skipped 10 previous similar messages <4>[1785716.163820] Lustre: atlas1-MDT0000: Client 210060d6-e38f-47de-2a59-0100583b806a (at 12485@gni100) reconnecting <4>[1785716.175666] Lustre: Skipped 36 previous similar messages <4>[1786231.927111] Lustre: atlas1-MDT0000: Client 4c0638cd-cbe7-481f-134f-30cf95159550 (at 16692@gni100) reconnecting <4>[1786231.938947] Lustre: Skipped 58 previous similar messages <4>[1786268.914440] Lustre: atlas1-MDT0000: haven't heard from client 231c4afe-ce96-e400-76ac-221f1fe206a2 (at 16666@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c646c00, cur 1458059350 expire 1458058450 last 1458057998 <4>[1786268.939817] Lustre: Skipped 11 previous similar messages <4>[1786861.026690] Lustre: atlas1-MDT0000: Client a95d0e9b-30f8-7b45-8b00-557fbd1e670e (at 14021@gni100) reconnecting <4>[1786861.038525] Lustre: Skipped 49 previous similar messages <4>[1787400.897697] Lustre: atlas1-MDT0000: haven't heard from client 6f0c0d0a-668f-10e7-adc7-009eade96db3 (at 10.38.145.143@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f42eda400, cur 1458060482 expire 1458059582 last 1458059130 <4>[1787400.923743] Lustre: Skipped 58 previous similar messages <4>[1794514.889576] Lustre: atlas1-MDT0000: haven't heard from client b6902dce-ada4-afcf-3d38-af3b533a3ff1 (at 18149@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f45bd6c00, cur 1458067596 expire 1458066696 last 1458066244 <4>[1799642.827899] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1799646.498105] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1799648.514540] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1800614.883404] Lustre: atlas1-MDT0000: haven't heard from client 2c4a0eeb-63b2-039d-ac66-b00b57e9d3af (at 4568@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f82301400, cur 1458073696 expire 1458072796 last 1458072344 <6>[1800722.396479] Lustre: atlas1-MDT0000-mdc-ffff883f37780000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1800722.396598] Lustre: client wants to enable acl, but mdt not! <6>[1800722.396599] Lustre: Layout lock feature supported. <6>[1800722.420235] Lustre: Skipped 1008 previous similar messages <4>[1800722.543540] Lustre: Mounted atlas1-client <4>[1803816.625422] Lustre: Unmounted atlas1-client <4>[1851483.829851] Lustre: atlas1-MDT0000: haven't heard from client 94308edc-01e4-d8e4-8e8e-3e2fe2e68f03 (at 16569@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c422400, cur 1458124565 expire 1458123665 last 1458123213 <4>[1851483.855247] Lustre: Skipped 5 previous similar messages <4>[1857246.918996] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1857250.939924] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1857252.976313] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1871644.699412] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1871648.316128] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1873212.035958] Lustre: atlas1-MDT0000: Client 066eadd6-f8d2-c930-780e-d089a3dc4afc (at 5852@gni100) reconnecting <4>[1873212.047686] Lustre: Skipped 6 previous similar messages <4>[1883871.797289] Lustre: atlas1-MDT0000: haven't heard from client 33cdc26b-9b82-eeff-73bd-39f7a283cd09 (at 7086@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e714000, cur 1458156953 expire 1458156053 last 1458155601 <4>[1883871.822573] Lustre: Skipped 1 previous similar message <4>[1886046.668123] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1886050.645860] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1886052.688442] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[1887121.961160] Lustre: atlas1-MDT0000-mdc-ffff883f467b9800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1887121.961298] Lustre: client wants to enable acl, but mdt not! <6>[1887121.961300] Lustre: Layout lock feature supported. <6>[1887121.984918] Lustre: Skipped 1008 previous similar messages <4>[1887122.101822] Lustre: Mounted atlas1-client <4>[1890251.508365] Lustre: Unmounted atlas1-client <4>[1921288.760446] Lustre: atlas1-MDT0000: haven't heard from client 9f720e73-a5f0-61fe-c984-da356d53ba11 (at 15893@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f79b52400, cur 1458194370 expire 1458193470 last 1458193018 <4>[1943643.283342] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1943646.923715] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1958042.538105] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1958046.538650] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1958048.547487] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1962239.721186] Lustre: atlas1-MDT0000: haven't heard from client e2055e74-5a5b-e8b8-744d-2f1845339af2 (at 12212@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4868e000, cur 1458235321 expire 1458234421 last 1458233969 <4>[1971238.713185] Lustre: atlas1-MDT0000: haven't heard from client 066eadd6-f8d2-c930-780e-d089a3dc4afc (at 5852@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e703c00, cur 1458244320 expire 1458243420 last 1458242968 <4>[1972444.846707] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[1972448.514929] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[1973522.291523] Lustre: atlas1-MDT0000-mdc-ffff88375e1ebc00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[1973522.291701] Lustre: client wants to enable acl, but mdt not! <6>[1973522.291703] Lustre: Layout lock feature supported. <6>[1973522.315279] Lustre: Skipped 1008 previous similar messages <4>[1973522.432859] Lustre: Mounted atlas1-client <4>[1976573.563324] Lustre: Unmounted atlas1-client <4>[2006848.677597] Lustre: atlas1-MDT0000: haven't heard from client 12dee4de-2d98-a73e-6d41-e489211be1d2 (at 10.38.144.39@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f33285800, cur 1458279930 expire 1458279030 last 1458278578 <4>[2008794.675259] Lustre: atlas1-MDT0000: haven't heard from client 52be8234-87cb-565d-e89e-c5e1f8f5a024 (at 7648@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e738400, cur 1458281876 expire 1458280976 last 1458280524 <4>[2030057.347760] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2030061.800863] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2030064.046752] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2058842.166510] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2058846.144919] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2058848.568549] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[2059922.074897] Lustre: atlas1-MDT0000-mdc-ffff883f7905bc00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2059922.075061] Lustre: client wants to enable acl, but mdt not! <6>[2059922.075063] Lustre: Layout lock feature supported. <6>[2059922.098658] Lustre: Skipped 1008 previous similar messages <4>[2059922.218236] Lustre: Mounted atlas1-client <4>[2062149.268878] Lustre: atlas1-MDT0000: Client d824e185-8b05-9c62-dd1b-c054c27317cf (at 9696@gni100) reconnecting <4>[2063110.542124] Lustre: Unmounted atlas1-client <4>[2063354.350891] Lustre: atlas1-MDT0000: Client f0f5575d-1089-7851-ab9b-b8170838c424 (at 7494@gni100) reconnecting <3>[2069685.877207] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 91@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff880ce19c5580/0xe00c22797b8e0ec6 lrc: 4/0,0 mode: PR/PR res: [0x2002ca13b:0x610c:0x0].0 bits 0x1b rrc: 4 type: IBT flags: 0x60200000000020 nid: 91@gni100 remote: 0x6097756385588912 expref: 357497 pid: 15517 timeout: 6364353765 lvb_type: 0 <3>[2069685.920334] LustreError: 15309:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 91@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff880ce19c5580/0xe00c22797b8e0ec6 lrc: 4/0,0 mode: PR/PR res: [0x2002ca13b:0x610c:0x0].0 bits 0x1b rrc: 4 type: IBT flags: 0x60200000000020 nid: 91@gni100 remote: 0x6097756385588912 expref: 357495 pid: 15517 timeout: 6364353765 lvb_type: 0 <4>[2071392.218427] Lustre: atlas1-MDT0000: Client f1f2c2d5-13c6-7c29-d46a-396d5ae92168 (at 2275@gni100) reconnecting <4>[2116445.290760] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2116449.425561] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2118873.563528] Lustre: atlas1-MDT0000: haven't heard from client 766236cd-a869-d312-36f7-fe449e52a22f (at 7922@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7b36a400, cur 1458391955 expire 1458391055 last 1458390603 <4>[2118873.588822] Lustre: Skipped 1 previous similar message <4>[2130850.583584] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2130858.423279] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2130863.094812] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2138387.497059] Lustre: atlas1-MDT0000: Client 3a66e0a6-bcd1-8f30-3227-3f73da251326 (at 2869@gni100) reconnecting <4>[2145256.434933] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2145261.752488] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2145264.425607] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2183193.179890] Lustre: atlas1-MDT0000: Client 35e8c6b5-935c-6e33-41db-d6ff833fbcbb (at 11373@gni100) reconnecting <6>[2200324.328704] Lustre: atlas1-MDT0000-mdc-ffff883f3a592000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2200324.328915] Lustre: client wants to enable acl, but mdt not! <6>[2200324.328917] Lustre: Layout lock feature supported. <6>[2200324.352462] Lustre: Skipped 1008 previous similar messages <4>[2200324.464497] Lustre: Mounted atlas1-client <4>[2202855.564656] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2202861.689268] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2202866.815265] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2204460.657564] Lustre: Unmounted atlas1-client <4>[2212570.462700] Lustre: atlas1-MDT0000: haven't heard from client da38e110-5a5e-b8c2-f97e-7bd06bd7f005 (at 14681@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7b836000, cur 1458485652 expire 1458484752 last 1458484300 <4>[2217258.964646] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2217266.104504] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2217269.880463] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2224761.450437] Lustre: atlas1-MDT0000: haven't heard from client 60f40853-070a-dae5-1999-a54879a3552c (at 7718@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e80cc00, cur 1458497843 expire 1458496943 last 1458496491 <4>[2228236.468757] Lustre: atlas1-MDT0000: Client a150fdbb-3a68-c19b-557d-00a0d125a37c (at 5906@gni100) reconnecting <4>[2231652.775114] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2231659.866201] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2231672.910683] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2288324.868771] Lustre: atlas1-MDT0000: Client cbf35652-8743-5616-0e9e-fb49a0b353cb (at 211@gni3) reconnecting <4>[2288325.443534] Lustre: atlas1-MDT0000: Client e18e2b64-8457-3c13-e564-ada031af344a (at 743@gni3) reconnecting <4>[2289261.805393] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2289281.341092] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2289293.802686] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2291235.932987] Lustre: atlas1-MDT0000: Client ef80c989-8efe-ac0d-21df-ef6d7ff3f8e5 (at 9244@gni100) reconnecting <4>[2291235.944716] Lustre: Skipped 1 previous similar message <3>[2291251.599893] LustreError: 138-a: atlas1-MDT0000: A client on nid 9229@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[2296570.073828] Lustre: atlas1-MDT0000: Client e18c08a0-d0f9-c42d-661b-b79f77f528b4 (at 3091@gni100) reconnecting <4>[2301981.365975] Lustre: atlas1-MDT0000: haven't heard from client 2658b4c7-dd39-8425-8de4-3dc23ca0555b (at 10.38.145.66@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f47ff1000, cur 1458575063 expire 1458574163 last 1458573711 <4>[2302432.365576] Lustre: atlas1-MDT0000: haven't heard from client a66121ff-07e9-6ccc-2b1b-5a6ec3c38c7c (at 9034@gni100) in 1241 seconds. I think it's dead, and I am evicting it. exp ffff883f7b8c6c00, cur 1458575514 expire 1458574614 last 1458574273 <4>[2303651.297703] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2303656.987245] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2303659.900285] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2303848.364217] Lustre: atlas1-MDT0000: haven't heard from client dcd6778f-0ef1-b413-c806-de7c36ac052e (at 10.38.144.28@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4824d400, cur 1458576930 expire 1458576030 last 1458575578 <4>[2304299.363896] Lustre: atlas1-MDT0000: haven't heard from client 37de141b-5ad4-66e5-7968-c1b5fe5f059b (at 10.38.144.26@o2ib5) in 1198 seconds. I think it's dead, and I am evicting it. exp ffff883f3f6d6000, cur 1458577381 expire 1458576481 last 1458576183 <4>[2304299.389868] Lustre: Skipped 2 previous similar messages <6>[2304724.049868] Lustre: atlas1-MDT0000-mdc-ffff883f30ffe800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2304724.050026] Lustre: client wants to enable acl, but mdt not! <6>[2304724.050028] Lustre: Layout lock feature supported. <6>[2304724.073642] Lustre: Skipped 1008 previous similar messages <4>[2304724.204798] Lustre: Mounted atlas1-client <4>[2308999.071034] Lustre: atlas1-MDT0000: Client e1b784a0-2558-0651-4711-25222e2130aa (at 6325@gni100) reconnecting <4>[2309401.946231] Lustre: Unmounted atlas1-client <4>[2312131.356888] Lustre: atlas1-MDT0000: haven't heard from client b0162eb6-0469-5fc9-cd28-9547830b69d7 (at 1579@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7cea8c00, cur 1458585213 expire 1458584313 last 1458583861 <4>[2318066.656855] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2318078.137125] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2318085.745289] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2344580.583489] Lustre: atlas1-MDT0000: Client 2dd4786d-cf4c-be00-50d2-1f3ae1cb0070 (at 696@gni3) reconnecting <4>[2344593.037375] Lustre: atlas1-MDT0000: Client 6e5942d2-e79d-656f-b694-f18d57e66d75 (at 683@gni3) reconnecting <4>[2344593.048810] Lustre: Skipped 4 previous similar messages <4>[2375657.657418] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2375670.950753] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2375674.568425] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[2382306.550042] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10.38.144.10@o2ib5 ns: mdt-atlas1-MDT0000_UUID lock: ffff8806f238e900/0xe00c22917e992124 lrc: 4/0,0 mode: PR/PR res: [0x20022295c:0x1bd08:0x0].0 bits 0x13 rrc: 12 type: IBT flags: 0x60200000000020 nid: 10.38.144.10@o2ib5 remote: 0xb6dca97f0c31ab42 expref: 1579 pid: 15933 timeout: 6676974749 lvb_type: 0 <3>[2382306.594831] LustreError: 15783:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 10.38.144.10@o2ib5) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8806f238e900/0xe00c22917e992124 lrc: 4/0,0 mode: PR/PR res: [0x20022295c:0x1bd08:0x0].0 bits 0x13 rrc: 12 type: IBT flags: 0x60200000000020 nid: 10.38.144.10@o2ib5 remote: 0xb6dca97f0c31ab42 expref: 1564 pid: 15933 timeout: 6676974749 lvb_type: 0 <4>[2385380.285103] Lustre: atlas1-MDT0000: haven't heard from client 11750af9-59ee-29d3-da12-fb47fca75f4e (at 10.38.144.24@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f688ee400, cur 1458658462 expire 1458657562 last 1458657110 <4>[2390042.381926] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2390046.399729] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2390048.898309] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[2391123.241491] Lustre: atlas1-MDT0000-mdc-ffff883f3e8cc800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2391123.241641] Lustre: client wants to enable acl, but mdt not! <6>[2391123.241643] Lustre: Layout lock feature supported. <6>[2391123.265255] Lustre: Skipped 1008 previous similar messages <4>[2391123.382508] Lustre: Mounted atlas1-client <4>[2394476.351692] Lustre: Unmounted atlas1-client <4>[2395412.275930] Lustre: atlas1-MDT0000: haven't heard from client 1243baaf-bf77-7213-8b6d-56fc2a881468 (at 10.36.247.149@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3dbc8c00, cur 1458668494 expire 1458667594 last 1458667142 <4>[2395863.275538] Lustre: atlas1-MDT0000: haven't heard from client 644128b5-066b-e70d-9521-d47add0d850b (at 10.36.202.185@o2ib) in 966 seconds. I think it's dead, and I am evicting it. exp ffff883f34d0b800, cur 1458668945 expire 1458668045 last 1458667979 <4>[2395863.301437] Lustre: Skipped 1 previous similar message <4>[2400073.271966] Lustre: atlas1-MDT0000: haven't heard from client 8acd0eef-a9be-0a4f-3a1e-c67b3a30005a (at 10.36.202.181@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f35735000, cur 1458673155 expire 1458672255 last 1458671803 <4>[2401622.591512] Lustre: atlas1-MDT0000: Client b23bff30-d25c-3ff7-347d-908c6005c877 (at 10.36.205.200@o2ib) reconnecting <4>[2401622.603926] Lustre: Skipped 1 previous similar message <4>[2404441.536457] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2404446.020326] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2404448.036428] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2423842.839159] Lustre: atlas1-MDT0000: Client 26090e2f-551c-5577-b34f-ddb1c9f57240 (at 10.36.226.122@o2ib) reconnecting <4>[2458622.675334] Lustre: atlas1-MDT0000: Client 41ec7ba1-36bf-2968-66ab-c5cf4aca856d (at 7657@gni100) reconnecting <4>[2462042.226470] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2462046.118924] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2465014.215153] Lustre: atlas1-MDT0000: haven't heard from client 9460b433-55d1-5a6a-473e-8e288af32171 (at 10694@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7b9e6000, cur 1458738096 expire 1458737196 last 1458736744 <4>[2465014.240537] Lustre: Skipped 7 previous similar messages <4>[2476442.203700] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2476445.849050] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[2477521.937630] Lustre: atlas1-MDT0000-mdc-ffff883f7dbc3000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2477521.937800] Lustre: client wants to enable acl, but mdt not! <6>[2477521.937801] Lustre: Layout lock feature supported. <6>[2477521.961388] Lustre: Skipped 1008 previous similar messages <4>[2477522.078116] Lustre: Mounted atlas1-client <4>[2480488.463067] Lustre: Unmounted atlas1-client <4>[2487166.194853] Lustre: atlas1-MDT0000: haven't heard from client 8e921114-898f-1e60-7f86-5d5ce3304507 (at 10.38.144.22@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3c070c00, cur 1458760248 expire 1458759348 last 1458758896 <4>[2490846.046244] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2490849.920299] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2491983.190262] Lustre: atlas1-MDT0000: haven't heard from client a974971d-d2d9-e966-1fdb-f1f2d82f6cdb (at 10.36.202.179@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f45dab800, cur 1458765065 expire 1458764165 last 1458763713 <3>[2494454.287883] LustreError: 138-a: atlas1-MDT0000: A client on nid 90@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[2500333.184295] Lustre: atlas1-MDT0000: haven't heard from client ec421a73-ffd3-6e11-fc9a-1763e8c17d20 (at 12327@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c9ca000, cur 1458773415 expire 1458772515 last 1458772063 <4>[2500333.209671] Lustre: Skipped 7 previous similar messages <3>[2501639.846007] LustreError: 138-a: atlas1-MDT0000: A client on nid 94@gni100 was evicted due to a lock blocking callback time out: rc -107 <3>[2505359.440409] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 90@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff882d51a6b480/0xe00c22a23a46c2ae lrc: 4/0,0 mode: PR/PR res: [0x2002cb821:0x27d6:0x0].0 bits 0x13 rrc: 16 type: IBT flags: 0x60200000000020 nid: 90@gni100 remote: 0xf68deb49f465d904 expref: 1075 pid: 15771 timeout: 6800027518 lvb_type: 0 <3>[2505359.483477] LustreError: 15837:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 90@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff882d51a6b480/0xe00c22a23a46c2ae lrc: 4/0,0 mode: PR/PR res: [0x2002cb821:0x27d6:0x0].0 bits 0x13 rrc: 15 type: IBT flags: 0x60200000000020 nid: 90@gni100 remote: 0xf68deb49f465d904 expref: 1047 pid: 15771 timeout: 6800027518 lvb_type: 0 <4>[2538455.148611] Lustre: atlas1-MDT0000: haven't heard from client 1489657c-61b3-8e65-73df-d16ad7fa98a9 (at 7980@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e3a8800, cur 1458811537 expire 1458810637 last 1458810185 <6>[2545921.939984] Lustre: atlas1-MDT0000-mdc-ffff883f47cc3400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2545921.940152] Lustre: client wants to enable acl, but mdt not! <6>[2545921.940153] Lustre: Layout lock feature supported. <6>[2545921.963794] Lustre: Skipped 1008 previous similar messages <4>[2545922.080095] Lustre: Mounted atlas1-client <4>[2548443.698955] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2548447.517211] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2548957.820359] Lustre: Unmounted atlas1-client <4>[2559024.130975] Lustre: atlas1-MDT0000: haven't heard from client 9a663dd5-aec5-ca3b-1465-4f6f9e2ff833 (at 10.36.202.185@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f474e9800, cur 1458832106 expire 1458831206 last 1458830754 <4>[2562057.208782] Lustre: atlas1-MDT0000: Client a8b53a08-1a1f-3635-75a8-84eb7704f58c (at 1153@gni100) reconnecting <4>[2562843.440281] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2562847.069769] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2564929.123418] Lustre: atlas1-MDT0000: haven't heard from client ab42d6d2-7c0f-c965-b605-26fd8faa3692 (at 10.36.202.185@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f46557400, cur 1458838011 expire 1458837111 last 1458836659 <4>[2572482.117489] Lustre: atlas1-MDT0000: haven't heard from client 50d63fa6-c6e1-dff1-a212-68fc5bb1c64b (at 10.36.202.185@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3b4d7800, cur 1458845564 expire 1458844664 last 1458844212 <4>[2574273.115037] Lustre: atlas1-MDT0000: haven't heard from client dca8d45d-3a22-4201-5783-d509cd619fef (at 10.36.202.185@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f476ff400, cur 1458847355 expire 1458846455 last 1458846003 <4>[2575478.113927] Lustre: atlas1-MDT0000: haven't heard from client 1a250b74-8006-e2bf-cd6b-6a3a75af6e8f (at 10.36.202.180@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f347edc00, cur 1458848560 expire 1458847660 last 1458847208 <4>[2575478.139914] Lustre: Skipped 7 previous similar messages <4>[2577247.361818] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2577251.265842] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2589702.208742] Lustre: atlas1-MDT0000: Client 26090e2f-551c-5577-b34f-ddb1c9f57240 (at 10.36.226.122@o2ib) reconnecting <4>[2592476.691760] Lustre: atlas1-MDT0000: Client 26090e2f-551c-5577-b34f-ddb1c9f57240 (at 10.36.226.122@o2ib) reconnecting <4>[2596766.079007] Lustre: atlas1-MDT0000: Client ff2ba532-2d47-0987-216d-074892bf29e5 (at 7635@gni100) reconnecting <6>[2632322.010681] Lustre: atlas1-MDT0000-mdc-ffff883f46736400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2632322.010815] Lustre: client wants to enable acl, but mdt not! <6>[2632322.010817] Lustre: Layout lock feature supported. <6>[2632322.034428] Lustre: Skipped 1008 previous similar messages <4>[2632322.182016] Lustre: Mounted atlas1-client <4>[2634845.765125] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2634849.787684] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2634851.818317] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2634872.045472] ptlrpcd_5: page allocation failure. order:1, mode:0x20 <4>[2634872.045591] ptlrpcd_6: page allocation failure. order:1, mode:0x20 <4>[2634872.045593] Pid: 14894, comm: ptlrpcd_6 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2634872.045594] Call Trace: <4>[2634872.045600] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2634872.045609] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2634872.045612] [] ? kmem_getpages+0x62/0x170 <4>[2634872.045613] [] ? fallback_alloc+0x1ba/0x270 <4>[2634872.045615] [] ? cache_grow+0x2cf/0x320 <4>[2634872.045616] ptlrpcd_1: page allocation failure. order:1, mode:0x20 <4>[2634872.045618] Pid: 14889, comm: ptlrpcd_1 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2634872.045620] [] ? ____cache_alloc_node+0x99/0x160 <4>[2634872.045620] Call Trace: <4>[2634872.045622] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2634872.045626] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2634872.045627] [] ? __kmalloc+0x199/0x230 <4>[2634872.045632] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2634872.045634] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2634872.045637] [] ? kmem_getpages+0x62/0x170 <4>[2634872.045638] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2634872.045640] [] ? fallback_alloc+0x1ba/0x270 <4>[2634872.045671] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2634872.045672] [] ? cache_grow+0x2cf/0x320 <4>[2634872.045690] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2634872.045691] [] ? ____cache_alloc_node+0x99/0x160 <4>[2634872.045708] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2634872.045710] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2634872.045726] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2634872.045728] [] ? __kmalloc+0x199/0x230 <4>[2634872.045730] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[2634872.045732] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2634872.045751] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2634872.045753] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2634872.045771] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2634872.045794] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2634872.045797] [] ? default_wake_function+0x0/0x20 <4>[2634872.045814] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2634872.045832] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2634872.045848] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2634872.045851] [] ? kthread+0x9e/0xc0 <4>[2634872.045867] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2634872.045869] [] ? child_rip+0xa/0x20 <4>[2634872.045887] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2634872.045889] [] ? kthread+0x0/0xc0 <4>[2634872.045907] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2634872.045908] [] ? child_rip+0x0/0x20 <6>[2634872.045909] Mem-Info: <4>[2634872.045911] [] ? default_wake_function+0x0/0x20 <4>[2634872.045912] Node 0 DMA per-cpu: <4>[2634872.045930] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2634872.045931] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2634872.045933] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2634872.045935] [] ? kthread+0x9e/0xc0 <4>[2634872.045935] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2634872.045936] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2634872.045938] [] ? child_rip+0xa/0x20 <4>[2634872.045939] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2634872.045940] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2634872.045941] [] ? kthread+0x0/0xc0 <4>[2634872.045942] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2634872.045943] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2634872.045944] [] ? child_rip+0x0/0x20 <4>[2634872.045945] Node 0 <6>[2634872.045946] Mem-Info: <4>[2634872.045946] DMA32 per-cpu: <4>[2634872.045947] Node 0 CPU 0: hi: 186, btch: 31 usd: 0 <4>[2634872.045948] DMA per-cpu: <4>[2634872.045949] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2634872.045950] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2634872.045951] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2634872.045951] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2634872.045952] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2634872.045953] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2634872.045954] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2634872.045954] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2634872.045955] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2634872.045956] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2634872.045957] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2634872.045958] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2634872.045958] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2634872.045959] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2634872.045960] Node 0 CPU 7: hi: 0, btch: 1 usd: 0 <4>[2634872.045961] Normal per-cpu: <4>[2634872.045961] Node 0 CPU 0: hi: 186, btch: 31 usd: 39 <4>[2634872.045963] DMA32 per-cpu: <4>[2634872.045963] CPU 1: hi: 186, btch: 31 usd: 24 <4>[2634872.045964] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2634872.045965] CPU 2: hi: 186, btch: 31 usd: 183 <4>[2634872.045966] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2634872.045967] CPU 3: hi: 186, btch: 31 usd: 149 <4>[2634872.045968] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2634872.045968] CPU 4: hi: 186, btch: 31 usd: 52 <4>[2634872.045969] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2634872.045970] CPU 5: hi: 186, btch: 31 usd: 58 <4>[2634872.045971] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2634872.045972] CPU 6: hi: 186, btch: 31 usd: 144 <4>[2634872.045972] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2634872.045973] CPU 7: hi: 186, btch: 31 usd: 31 <4>[2634872.045974] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2634872.045976] active_anon:4855514 inactive_anon:166710 isolated_anon:0 <4>[2634872.045976] active_file:8051933 inactive_file:24456089 isolated_file:0 <4>[2634872.045977] unevictable:8917 dirty:1115 writeback:0 unstable:0 <4>[2634872.045977] free:125549 slab_reclaimable:13453278 slab_unreclaimable:14087733 <4>[2634872.045978] mapped:8470 shmem:241185 pagetables:10664 bounce:0 <4>[2634872.045979] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2634872.045979] Node 0 Node 0 Normal per-cpu: <4>[2634872.045982] DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2634872.045984] CPU 0: hi: 186, btch: 31 usd: 39 <4>[2634872.045984] lowmem_reserve[]:CPU 1: hi: 186, btch: 31 usd: 24 <4>[2634872.045985] 0CPU 2: hi: 186, btch: 31 usd: 183 <4>[2634872.045987] 1880CPU 3: hi: 186, btch: 31 usd: 149 <4>[2634872.045988] 258420CPU 4: hi: 186, btch: 31 usd: 52 <4>[2634872.045989] 258420CPU 5: hi: 186, btch: 31 usd: 58 <4>[2634872.045990] <4>[2634872.045991] CPU 6: hi: 186, btch: 31 usd: 144 <4>[2634872.045991] Node 0 CPU 7: hi: 186, btch: 31 usd: 31 <4>[2634872.045994] DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2634872.045997] active_anon:4855514 inactive_anon:166710 isolated_anon:0 <4>[2634872.045998] active_file:8051933 inactive_file:24456089 isolated_file:0 <4>[2634872.045998] unevictable:8917 dirty:1115 writeback:0 unstable:0 <4>[2634872.045998] free:125549 slab_reclaimable:13453278 slab_unreclaimable:14087733 <4>[2634872.045999] mapped:8470 shmem:241185 pagetables:10664 bounce:0 <4>[2634872.045999] lowmem_reserve[]:Node 0 0DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2634872.046004] 0lowmem_reserve[]: 256540 0 256540 1880 <4>[2634872.046006] 258420Node 0 258420Normal free:101340kB min:67088kB low:83860kB high:100632kB active_anon:19422056kB inactive_anon:666840kB active_file:32207732kB inactive_file:97824356kB unevictable:35668kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:4460kB writeback:0kB mapped:33880kB shmem:964740kB slab_reclaimable:53813112kB slab_unreclaimable:56350932kB kernel_stack:74352kB pagetables:42656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2634872.046011] <4>[2634872.046012] lowmem_reserve[]:Node 0 0DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2634872.046016] 0lowmem_reserve[]: 0 0 0 0 <4>[2634872.046018] 256540Node 0 256540DMA: <4>[2634872.046020] 1*4kB Node 0 1*8kB Normal free:101340kB min:67088kB low:83860kB high:100632kB active_anon:19422056kB inactive_anon:666840kB active_file:32207732kB inactive_file:97824356kB unevictable:35668kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:4460kB writeback:0kB mapped:33880kB shmem:964740kB slab_reclaimable:53813112kB slab_unreclaimable:56350932kB kernel_stack:74352kB pagetables:42656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2634872.046025] 1*16kB lowmem_reserve[]:1*32kB 01*64kB 00*128kB 01*256kB 00*512kB <4>[2634872.046029] 1*1024kB Node 0 1*2048kB DMA: 3*4096kB 1*4kB = 15740kB <4>[2634872.046032] 1*8kB Node 0 1*16kB DMA32: 1*32kB 9*4kB 1*64kB 3*8kB 0*128kB 4*16kB 1*256kB 7*32kB 0*512kB 6*64kB 1*1024kB 5*128kB 1*2048kB 7*256kB 3*4096kB 10*512kB = 15740kB <4>[2634872.046040] 8*1024kB Node 0 4*2048kB DMA32: 88*4096kB 9*4kB = 385116kB <4>[2634872.046043] 3*8kB Node 0 4*16kB Normal: 7*32kB 25312*4kB 6*64kB 0*8kB 5*128kB 0*16kB 7*256kB 0*32kB 10*512kB 0*64kB 8*1024kB 0*128kB 4*2048kB 0*256kB 88*4096kB 0*512kB = 385116kB <4>[2634872.046051] 0*1024kB Node 0 0*2048kB Normal: 0*4096kB 25312*4kB = 101248kB <4>[2634872.046053] 0*8kB 32750393 total pagecache pages <4>[2634872.046055] 0*16kB 0 pages in swap cache <4>[2634872.046056] 0*32kB Swap cache stats: add 0, delete 0, find 0/0 <4>[2634872.046057] 0*64kB Free swap = 0kB <4>[2634872.046058] 0*128kB Total swap = 0kB <4>[2634872.046059] 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 101248kB <4>[2634872.046060] 32750393 total pagecache pages <4>[2634872.046061] 0 pages in swap cache <4>[2634872.046061] Swap cache stats: add 0, delete 0, find 0/0 <4>[2634872.046062] Free swap = 0kB <4>[2634872.046062] Total swap = 0kB <6>[2634872.380678] 67108863 pages RAM <6>[2634872.380679] 1015653 pages reserved <6>[2634872.380680] 28182064 pages shared <6>[2634872.380680] 37483494 pages non-shared <3>[2634872.380872] LustreError: 14894:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2634872.380874] LustreError: 14894:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1484006800 total bytes allocated by lnet <6>[2634872.712289] 67108863 pages RAM <6>[2634872.712290] 1015653 pages reserved <6>[2634872.712290] 28221609 pages shared <6>[2634872.712291] 37482479 pages non-shared <4>[2634873.310984] Pid: 14893, comm: ptlrpcd_5 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2634873.320565] Call Trace: <4>[2634873.323714] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2634873.331435] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2634873.340054] [] ? kmem_getpages+0x62/0x170 <4>[2634873.346795] [] ? fallback_alloc+0x1ba/0x270 <4>[2634873.353731] [] ? cache_grow+0x2cf/0x320 <4>[2634873.360280] [] ? ____cache_alloc_node+0x99/0x160 <4>[2634873.367708] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2634873.376322] [] ? __kmalloc+0x199/0x230 <4>[2634873.382778] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2634873.391390] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2634873.399527] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2634873.407165] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2634873.415285] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2634873.423816] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2634873.431941] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2634873.439682] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2634873.446815] [] ? default_wake_function+0x0/0x20 <4>[2634873.454159] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2634873.461097] [] ? kthread+0x9e/0xc0 <4>[2634873.467161] [] ? child_rip+0xa/0x20 <4>[2634873.473320] [] ? kthread+0x0/0xc0 <4>[2634873.479281] [] ? child_rip+0x0/0x20 <6>[2634873.485527] Mem-Info: <4>[2634873.488481] Node 0 DMA per-cpu: <4>[2634873.492408] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2634873.498180] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2634873.503941] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2634873.509720] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2634873.515488] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2634873.521258] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2634873.527031] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2634873.532799] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2634873.538567] Node 0 DMA32 per-cpu: <4>[2634873.542699] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2634873.548469] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2634873.554238] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2634873.560011] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2634873.565790] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2634873.571556] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2634873.577324] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2634873.583093] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2634873.588861] Node 0 Normal per-cpu: <4>[2634873.593088] CPU 0: hi: 186, btch: 31 usd: 172 <4>[2634873.598854] CPU 1: hi: 186, btch: 31 usd: 173 <4>[2634873.604635] CPU 2: hi: 186, btch: 31 usd: 87 <4>[2634873.610403] CPU 3: hi: 186, btch: 31 usd: 156 <4>[2634873.616173] CPU 4: hi: 186, btch: 31 usd: 137 <4>[2634873.621945] CPU 5: hi: 186, btch: 31 usd: 58 <4>[2634873.627717] CPU 6: hi: 186, btch: 31 usd: 82 <4>[2634873.633498] CPU 7: hi: 186, btch: 31 usd: 147 <4>[2634873.639270] active_anon:4855611 inactive_anon:166710 isolated_anon:0 <4>[2634873.639271] active_file:8051938 inactive_file:24447497 isolated_file:0 <4>[2634873.639271] unevictable:8917 dirty:1349 writeback:129 unstable:0 <4>[2634873.639272] free:139927 slab_reclaimable:13450217 slab_unreclaimable:14084284 <4>[2634873.639272] mapped:8470 shmem:241185 pagetables:10664 bounce:0 <4>[2634873.677565] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2634873.719624] lowmem_reserve[]: 0 1880 258420 258420 <4>[2634873.725459] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2634873.768567] lowmem_reserve[]: 0 0 256540 256540 <4>[2634873.774113] Node 0 Normal free:158852kB min:67088kB low:83860kB high:100632kB active_anon:19422444kB inactive_anon:666840kB active_file:32207752kB inactive_file:97789988kB unevictable:35668kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:5396kB writeback:516kB mapped:33880kB shmem:964740kB slab_reclaimable:53800868kB slab_unreclaimable:56337136kB kernel_stack:74352kB pagetables:42656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2634873.824765] lowmem_reserve[]: 0 0 0 0 <4>[2634873.829351] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2634873.842013] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2634873.855170] Node 0 Normal: 29763*4kB 313*8kB 57*16kB 46*32kB 79*64kB 221*128kB 3*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 159076kB <4>[2634873.874612] 32741728 total pagecache pages <4>[2634873.879603] 0 pages in swap cache <4>[2634873.883719] Swap cache stats: add 0, delete 0, find 0/0 <4>[2634873.889978] Free swap = 0kB <4>[2634873.893618] Total swap = 0kB <6>[2634874.231829] 67108863 pages RAM <6>[2634874.235657] 1015653 pages reserved <6>[2634874.239882] 28220495 pages shared <6>[2634874.244009] 37478587 pages non-shared <3>[2634874.248678] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2634874.273410] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <3>[2634874.285234] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1483621632 total bytes allocated by lnet <3>[2634874.298225] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <4>[2635172.045343] ptlrpcd_7: page allocation failure. order:1, mode:0x20 <4>[2635172.045684] ptlrpcd_2: page allocation failure. order:1, mode:0x20 <4>[2635172.045686] Pid: 14890, comm: ptlrpcd_2 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2635172.045687] Call Trace: <4>[2635172.045692] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2635172.045702] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2635172.045705] [] ? kmem_getpages+0x62/0x170 <4>[2635172.045707] [] ? fallback_alloc+0x1ba/0x270 <4>[2635172.045708] [] ? cache_grow+0x2cf/0x320 <4>[2635172.045709] [] ? ____cache_alloc_node+0x99/0x160 <4>[2635172.045712] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2635172.045713] [] ? __kmalloc+0x199/0x230 <4>[2635172.045715] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2635172.045716] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2635172.045751] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2635172.045768] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2635172.045785] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2635172.045802] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2635172.045804] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[2635172.045823] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2635172.045841] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2635172.045844] [] ? default_wake_function+0x0/0x20 <4>[2635172.045862] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2635172.045864] [] ? kthread+0x9e/0xc0 <4>[2635172.045866] [] ? child_rip+0xa/0x20 <4>[2635172.045868] [] ? kthread+0x0/0xc0 <4>[2635172.045869] [] ? child_rip+0x0/0x20 <6>[2635172.045870] Mem-Info: <4>[2635172.045870] Node 0 DMA per-cpu: <4>[2635172.045872] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2635172.045872] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2635172.045873] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2635172.045874] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2635172.045875] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2635172.045875] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2635172.045876] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2635172.045877] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2635172.045877] Node 0 DMA32 per-cpu: <4>[2635172.045878] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2635172.045879] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2635172.045879] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2635172.045880] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2635172.045881] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2635172.045881] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2635172.045882] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2635172.045882] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2635172.045883] Node 0 Normal per-cpu: <4>[2635172.045884] CPU 0: hi: 186, btch: 31 usd: 60 <4>[2635172.045884] CPU 1: hi: 186, btch: 31 usd: 152 <4>[2635172.045885] CPU 2: hi: 186, btch: 31 usd: 185 <4>[2635172.045886] CPU 3: hi: 186, btch: 31 usd: 33 <4>[2635172.045886] CPU 4: hi: 186, btch: 31 usd: 32 <4>[2635172.045887] CPU 5: hi: 186, btch: 31 usd: 51 <4>[2635172.045887] CPU 6: hi: 186, btch: 31 usd: 144 <4>[2635172.045888] CPU 7: hi: 186, btch: 31 usd: 143 <4>[2635172.045890] active_anon:4855644 inactive_anon:166713 isolated_anon:0 <4>[2635172.045891] active_file:8054425 inactive_file:25368926 isolated_file:0 <4>[2635172.045891] unevictable:8917 dirty:2185 writeback:0 unstable:0 <4>[2635172.045891] free:121762 slab_reclaimable:12790635 slab_unreclaimable:13836771 <4>[2635172.045892] mapped:8470 shmem:241220 pagetables:10664 bounce:0 <4>[2635172.045892] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2635172.045896] lowmem_reserve[]: 0 1880 258420 258420 <4>[2635172.045897] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2635172.045901] lowmem_reserve[]: 0 0 256540 256540 <4>[2635172.045902] Node 0 Normal free:86192kB min:67088kB low:83860kB high:100632kB active_anon:19422576kB inactive_anon:666852kB active_file:32217700kB inactive_file:101475704kB unevictable:35668kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:8740kB writeback:0kB mapped:33880kB shmem:964880kB slab_reclaimable:51162540kB slab_unreclaimable:55347084kB kernel_stack:74368kB pagetables:42656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2635172.045906] lowmem_reserve[]: 0 0 0 0 <4>[2635172.045906] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2635172.045909] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2635172.045912] Node 0 Normal: 21328*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 85312kB <4>[2635172.045915] 33665614 total pagecache pages <4>[2635172.045916] 0 pages in swap cache <4>[2635172.045916] Swap cache stats: add 0, delete 0, find 0/0 <4>[2635172.045917] Free swap = 0kB <4>[2635172.045917] Total swap = 0kB <6>[2635172.383783] 67108863 pages RAM <6>[2635172.383784] 1015653 pages reserved <6>[2635172.383785] 29324553 pages shared <6>[2635172.383785] 36554190 pages non-shared <3>[2635172.383990] LustreError: 14890:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2635172.383993] LustreError: 14890:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1483950376 total bytes allocated by lnet <4>[2635172.701033] Pid: 14895, comm: ptlrpcd_7 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2635172.710644] Call Trace: <4>[2635172.713803] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2635172.721534] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2635172.730162] [] ? kmem_getpages+0x62/0x170 <4>[2635172.736909] [] ? fallback_alloc+0x1ba/0x270 <4>[2635172.743844] [] ? cache_grow+0x2cf/0x320 <4>[2635172.750394] [] ? ____cache_alloc_node+0x99/0x160 <4>[2635172.757815] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2635172.766436] [] ? __kmalloc+0x199/0x230 <4>[2635172.772886] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2635172.781515] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2635172.789666] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2635172.797309] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2635172.805424] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2635172.813963] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2635172.822105] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2635172.829835] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2635172.836967] [] ? default_wake_function+0x0/0x20 <4>[2635172.844332] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2635172.851280] [] ? kthread+0x9e/0xc0 <4>[2635172.857352] [] ? child_rip+0xa/0x20 <4>[2635172.863517] [] ? kthread+0x0/0xc0 <4>[2635172.869584] [] ? child_rip+0x0/0x20 <6>[2635172.875777] Mem-Info: <4>[2635172.878730] Node 0 DMA per-cpu: <4>[2635172.882667] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2635172.888446] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2635172.894214] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2635172.899983] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2635172.905753] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2635172.911543] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2635172.917323] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2635172.923093] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2635172.928863] Node 0 DMA32 per-cpu: <4>[2635172.932993] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2635172.938792] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2635172.944582] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2635172.950366] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2635172.956135] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2635172.961915] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2635172.967710] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2635172.973501] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2635172.979291] Node 0 Normal per-cpu: <4>[2635172.983554] CPU 0: hi: 186, btch: 31 usd: 181 <4>[2635172.989334] CPU 1: hi: 186, btch: 31 usd: 140 <4>[2635173.000551] CPU 2: hi: 186, btch: 31 usd: 157 <4>[2635173.006331] CPU 3: hi: 186, btch: 31 usd: 61 <4>[2635173.012109] CPU 4: hi: 186, btch: 31 usd: 36 <4>[2635173.017898] CPU 5: hi: 186, btch: 31 usd: 51 <4>[2635173.023671] CPU 6: hi: 186, btch: 31 usd: 161 <4>[2635173.029452] CPU 7: hi: 186, btch: 31 usd: 140 <4>[2635173.035221] active_anon:4855547 inactive_anon:166713 isolated_anon:0 <4>[2635173.035221] active_file:8054427 inactive_file:25362787 isolated_file:0 <4>[2635173.035222] unevictable:8917 dirty:2049 writeback:0 unstable:0 <4>[2635173.035222] free:132299 slab_reclaimable:12788828 slab_unreclaimable:13834151 <4>[2635173.035222] mapped:8470 shmem:241220 pagetables:10664 bounce:0 <4>[2635173.073389] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2635173.115581] lowmem_reserve[]: 0 1880 258420 258420 <4>[2635173.121412] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2635173.164543] lowmem_reserve[]: 0 0 256540 256540 <4>[2635173.170114] Node 0 Normal free:128340kB min:67088kB low:83860kB high:100632kB active_anon:19422188kB inactive_anon:666852kB active_file:32217708kB inactive_file:101451148kB unevictable:35668kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:8196kB writeback:0kB mapped:33880kB shmem:964880kB slab_reclaimable:51155312kB slab_unreclaimable:55336604kB kernel_stack:74352kB pagetables:42656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2635173.220674] lowmem_reserve[]: 0 0 0 0 <4>[2635173.225272] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2635173.237945] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2635173.251096] Node 0 Normal: 24393*4kB 416*8kB 181*16kB 113*32kB 69*64kB 52*128kB 29*256kB 4*512kB 0*1024kB 0*2048kB 0*4096kB = 127956kB <4>[2635173.265423] 33659523 total pagecache pages <4>[2635173.270428] 0 pages in swap cache <4>[2635173.274560] Swap cache stats: add 0, delete 0, find 0/0 <4>[2635173.280828] Free swap = 0kB <4>[2635173.284464] Total swap = 0kB <6>[2635173.633042] 67108863 pages RAM <6>[2635173.636867] 1015653 pages reserved <6>[2635173.641101] 29154658 pages shared <6>[2635173.645213] 36554585 pages non-shared <4>[2635411.126305] Lustre: Unmounted atlas1-client <4>[2649238.554174] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2649242.239094] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2649244.263410] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2660863.514306] Lustre: atlas1-MDT0000: Client 2a4fda27-71d8-a7a7-2659-75121837df26 (at 9318@gni100) reconnecting <4>[2663644.782222] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2663648.437305] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[2665624.357985] LustreError: 138-a: atlas1-MDT0000: A client on nid 95@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[2704869.649735] Lustre: atlas1-MDT0000: Client f1ee2bb4-c228-dcda-2f8f-e5f2c513d4e3 (at 12235@gni100) reconnecting <3>[2717445.567609] LustreError: 138-a: atlas1-MDT0000: A client on nid 9299@gni100 was evicted due to a lock blocking callback time out: rc -107 <3>[2717445.582068] LustreError: Skipped 1 previous similar message <6>[2718721.225591] Lustre: atlas1-MDT0000-mdc-ffff883f398f6000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2718721.225771] Lustre: client wants to enable acl, but mdt not! <6>[2718721.225772] Lustre: Layout lock feature supported. <6>[2718721.249335] Lustre: Skipped 1008 previous similar messages <4>[2718721.367893] Lustre: Mounted atlas1-client <4>[2720971.259501] ptlrpcd_7: page allocation failure. order:1, mode:0x20 <4>[2720971.260459] ptlrpcd_1: page allocation failure. order:1, mode:0x20 <4>[2720971.260461] Pid: 14889, comm: ptlrpcd_1 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2720971.260462] Call Trace: <4>[2720971.260468] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2720971.260477] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2720971.260480] [] ? kmem_getpages+0x62/0x170 <4>[2720971.260482] [] ? fallback_alloc+0x1ba/0x270 <4>[2720971.260483] [] ? cache_grow+0x2cf/0x320 <4>[2720971.260484] [] ? ____cache_alloc_node+0x99/0x160 <4>[2720971.260486] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2720971.260488] [] ? __kmalloc+0x199/0x230 <4>[2720971.260490] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2720971.260491] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2720971.260525] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2720971.260542] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2720971.260559] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2720971.260576] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2720971.260595] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2720971.260613] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2720971.260616] [] ? default_wake_function+0x0/0x20 <4>[2720971.260634] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2720971.260636] [] ? kthread+0x9e/0xc0 <4>[2720971.260638] [] ? child_rip+0xa/0x20 <4>[2720971.260640] [] ? kthread+0x0/0xc0 <4>[2720971.260641] [] ? child_rip+0x0/0x20 <6>[2720971.260642] Mem-Info: <4>[2720971.260642] Node 0 DMA per-cpu: <4>[2720971.260644] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2720971.260644] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2720971.260645] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2720971.260646] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2720971.260646] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2720971.260647] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2720971.260648] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2720971.260648] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2720971.260649] Node 0 DMA32 per-cpu: <4>[2720971.260650] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2720971.260650] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2720971.260651] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2720971.260652] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2720971.260652] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2720971.260653] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2720971.260654] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2720971.260654] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2720971.260655] Node 0 Normal per-cpu: <4>[2720971.260655] CPU 0: hi: 186, btch: 31 usd: 84 <4>[2720971.260656] CPU 1: hi: 186, btch: 31 usd: 49 <4>[2720971.260657] CPU 2: hi: 186, btch: 31 usd: 169 <4>[2720971.260657] CPU 3: hi: 186, btch: 31 usd: 49 <4>[2720971.260658] CPU 4: hi: 186, btch: 31 usd: 162 <4>[2720971.260659] CPU 5: hi: 186, btch: 31 usd: 52 <4>[2720971.260659] CPU 6: hi: 186, btch: 31 usd: 76 <4>[2720971.260660] CPU 7: hi: 186, btch: 31 usd: 166 <4>[2720971.260662] active_anon:4838774 inactive_anon:166079 isolated_anon:0 <4>[2720971.260662] active_file:10606671 inactive_file:23649916 isolated_file:0 <4>[2720971.260663] unevictable:7670 dirty:991 writeback:0 unstable:0 <4>[2720971.260663] free:685534 slab_reclaimable:12402310 slab_unreclaimable:12846672 <4>[2720971.260664] mapped:8498 shmem:222059 pagetables:10664 bounce:0 <4>[2720971.260664] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2720971.260668] lowmem_reserve[]: 0 1880 258420 258420 <4>[2720971.260669] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2720971.260672] lowmem_reserve[]: 0 0 256540 256540 <4>[2720971.260673] Node 0 Normal free:2341280kB min:67088kB low:83860kB high:100632kB active_anon:19355096kB inactive_anon:664316kB active_file:42426684kB inactive_file:94599664kB unevictable:30680kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:3964kB writeback:0kB mapped:33992kB shmem:888236kB slab_reclaimable:49609240kB slab_unreclaimable:51386688kB kernel_stack:74352kB pagetables:42656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2720971.260677] lowmem_reserve[]: 0 0 0 0 <4>[2720971.260678] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2720971.260681] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2720971.260684] Node 0 Normal: 585219*4kB 16*8kB 7*16kB 4*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2341308kB <4>[2720971.260687] 34479769 total pagecache pages <4>[2720971.260688] 0 pages in swap cache <4>[2720971.260688] Swap cache stats: add 0, delete 0, find 0/0 <4>[2720971.260689] Free swap = 0kB <4>[2720971.260689] Total swap = 0kB <6>[2720971.604238] 67108863 pages RAM <6>[2720971.604240] 1015653 pages reserved <6>[2720971.604240] 30274387 pages shared <6>[2720971.604241] 34865473 pages non-shared <3>[2720971.604325] LustreError: 14889:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2720971.604327] LustreError: 14889:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <3>[2720971.604329] LustreError: 14889:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1483795488 total bytes allocated by lnet <3>[2720971.604330] LustreError: 14889:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <4>[2720971.936494] Pid: 14895, comm: ptlrpcd_7 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2720971.946076] Call Trace: <4>[2720971.949226] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2720971.956939] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2720971.965559] [] ? kmem_getpages+0x62/0x170 <4>[2720971.972294] [] ? fallback_alloc+0x1ba/0x270 <4>[2720971.979232] [] ? cache_grow+0x2cf/0x320 <4>[2720971.985780] [] ? ____cache_alloc_node+0x99/0x160 <4>[2720971.993209] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2720972.001812] [] ? __kmalloc+0x199/0x230 <4>[2720972.008263] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2720972.016878] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2720972.025011] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2720972.032650] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2720972.040773] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2720972.049310] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2720972.057410] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[2720972.064851] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2720972.072585] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2720972.079717] [] ? default_wake_function+0x0/0x20 <4>[2720972.087065] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2720972.094004] [] ? kthread+0x9e/0xc0 <4>[2720972.100067] [] ? child_rip+0xa/0x20 <4>[2720972.106228] [] ? kthread+0x0/0xc0 <4>[2720972.112191] [] ? child_rip+0x0/0x20 <6>[2720972.118349] Mem-Info: <4>[2720972.121304] Node 0 DMA per-cpu: <4>[2720972.125242] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2720972.131010] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2720972.136783] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2720972.142552] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2720972.148330] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2720972.154195] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2720972.159964] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2720972.165735] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2720972.171507] Node 0 DMA32 per-cpu: <4>[2720972.175640] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2720972.181408] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2720972.187178] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2720972.192947] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2720972.198716] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2720972.204490] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2720972.210260] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2720972.216030] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2720972.221798] Node 0 Normal per-cpu: <4>[2720972.226025] CPU 0: hi: 186, btch: 31 usd: 68 <4>[2720972.231795] CPU 1: hi: 186, btch: 31 usd: 144 <4>[2720972.237566] CPU 2: hi: 186, btch: 31 usd: 181 <4>[2720972.243335] CPU 3: hi: 186, btch: 31 usd: 58 <4>[2720972.249105] CPU 4: hi: 186, btch: 31 usd: 162 <4>[2720972.254874] CPU 5: hi: 186, btch: 31 usd: 67 <4>[2720972.260647] CPU 6: hi: 186, btch: 31 usd: 59 <4>[2720972.266417] CPU 7: hi: 186, btch: 31 usd: 179 <4>[2720972.272188] active_anon:4838774 inactive_anon:166079 isolated_anon:0 <4>[2720972.272188] active_file:10606671 inactive_file:23646361 isolated_file:0 <4>[2720972.272189] unevictable:7670 dirty:1000 writeback:0 unstable:0 <4>[2720972.272189] free:692651 slab_reclaimable:12400636 slab_unreclaimable:12844711 <4>[2720972.272190] mapped:8498 shmem:222059 pagetables:10664 bounce:0 <4>[2720972.310371] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2720972.352428] lowmem_reserve[]: 0 1880 258420 258420 <4>[2720972.358261] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2720972.401468] lowmem_reserve[]: 0 0 256540 256540 <4>[2720972.407019] Node 0 Normal free:2369748kB min:67088kB low:83860kB high:100632kB active_anon:19355096kB inactive_anon:664316kB active_file:42426684kB inactive_file:94585444kB unevictable:30680kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:4000kB writeback:0kB mapped:33992kB shmem:888236kB slab_reclaimable:49602544kB slab_unreclaimable:51378844kB kernel_stack:74352kB pagetables:42656kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2720972.457560] lowmem_reserve[]: 0 0 0 0 <4>[2720972.462131] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2720972.474778] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2720972.487920] Node 0 Normal: 588511*4kB 85*8kB 1*16kB 0*32kB 219*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2368756kB <4>[2720972.501836] 34476180 total pagecache pages <4>[2720972.506827] 0 pages in swap cache <4>[2720972.510943] Swap cache stats: add 0, delete 0, find 0/0 <4>[2720972.517200] Free swap = 0kB <4>[2720972.520832] Total swap = 0kB <6>[2720972.868116] 67108863 pages RAM <6>[2720972.871948] 1015653 pages reserved <6>[2720972.876163] 30271331 pages shared <6>[2720972.880279] 34865780 pages non-shared <3>[2720972.884987] LustreError: 14895:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2720972.909718] LustreError: 14895:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1483769728 total bytes allocated by lnet <4>[2721246.994013] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2721250.923318] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2721253.090995] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2721902.010170] Lustre: Unmounted atlas1-client <4>[2729772.965709] Lustre: atlas1-MDT0000: haven't heard from client abcfeee7-20b4-70ac-2b4f-c679301078c8 (at 9310@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3f7ed400, cur 1459002855 expire 1459001955 last 1459001503 <4>[2729772.991001] Lustre: Skipped 7 previous similar messages <4>[2732088.960093] Lustre: atlas1-MDT0000: haven't heard from client c73cc88a-ad37-d721-820d-b306b38b8be5 (at 6202@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a1afc00, cur 1459005171 expire 1459004271 last 1459003819 <4>[2735639.152360] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2735642.997737] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2735645.127349] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2737958.397108] Lustre: atlas1-MDT0000: Client 5bdd467b-bfd2-ea2b-d5f2-ae600ac330f5 (at 9190@gni100) reconnecting <4>[2747954.945050] Lustre: atlas1-MDT0000: haven't heard from client 62899fd8-d695-97b4-377d-20ac3406a73e (at 3337@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a0a6400, cur 1459021037 expire 1459020137 last 1459019685 <4>[2747954.970331] Lustre: Skipped 1 previous similar message <4>[2750040.171431] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2750043.867545] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2780223.441883] Lustre: atlas1-MDT0000: Client d7ebc1f6-1da4-9f1d-5b4f-99c360e5e2b0 (at 11434@gni100) reconnecting <4>[2801225.324663] Lustre: atlas1-MDT0000: Client 1bfa5cc1-69f6-8776-0bcf-3be42d45dd94 (at 5468@gni100) reconnecting <6>[2805121.018680] Lustre: atlas1-MDT0000-mdc-ffff883f7d9eb400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2805121.018791] Lustre: client wants to enable acl, but mdt not! <6>[2805121.018793] Lustre: Layout lock feature supported. <6>[2805121.042440] Lustre: Skipped 1008 previous similar messages <4>[2805121.192744] Lustre: Mounted atlas1-client <4>[2806621.052472] ptlrpcd_2: page allocation failure. order:1, mode:0x20 <4>[2806621.052475] ptlrpcd_3: page allocation failure. order:1, mode:0x20 <4>[2806621.052477] Pid: 14891, comm: ptlrpcd_3 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2806621.052478] Call Trace: <4>[2806621.052483] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2806621.052492] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2806621.052496] [] ? kmem_getpages+0x62/0x170 <4>[2806621.052497] [] ? fallback_alloc+0x1ba/0x270 <4>[2806621.052498] [] ? cache_grow+0x2cf/0x320 <4>[2806621.052499] [] ? ____cache_alloc_node+0x99/0x160 <4>[2806621.052502] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806621.052503] [] ? __kmalloc+0x199/0x230 <4>[2806621.052505] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806621.052506] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2806621.052540] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2806621.052557] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2806621.052559] [] ? thread_return+0x4e/0x7d0 <4>[2806621.052576] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2806621.052578] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[2806621.052597] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2806621.052599] ptlrpcd_0: page allocation failure. order:1, mode:0x20 <4>[2806621.052617] [] ? ptlrpcd+0x20b/0x370 [ptlrpc] <4>[2806621.052618] Pid: 14888, comm: ptlrpcd_0 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2806621.052620] Call Trace: <4>[2806621.052623] [] ? default_wake_function+0x0/0x20 <4>[2806621.052626] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2806621.052643] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2806621.052648] [] ? kiblnd_send+0x2a3/0x9b0 [ko2iblnd] <4>[2806621.052651] [] ? kthread+0x9e/0xc0 <4>[2806621.052653] [] ? kmem_getpages+0x62/0x170 <4>[2806621.052655] [] ? child_rip+0xa/0x20 <4>[2806621.052657] [] ? fallback_alloc+0x1ba/0x270 <4>[2806621.052659] [] ? kthread+0x0/0xc0 <4>[2806621.052660] [] ? cache_grow+0x2cf/0x320 <4>[2806621.052662] [] ? child_rip+0x0/0x20 <6>[2806621.052662] Mem-Info: <4>[2806621.052664] [] ? ____cache_alloc_node+0x99/0x160 <4>[2806621.052665] Node 0 DMA per-cpu: <4>[2806621.052667] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806621.052668] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2806621.052669] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2806621.052671] [] ? __kmalloc+0x199/0x230 <4>[2806621.052672] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2806621.052673] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2806621.052674] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806621.052675] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2806621.052676] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2806621.052678] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2806621.052679] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2806621.052680] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2806621.052701] [] ? after_reply+0x32a/0xec0 [ptlrpc] <4>[2806621.052702] Node 0 DMA32 per-cpu: <4>[2806621.052704] [] ? thread_return+0x4e/0x7d0 <4>[2806621.052705] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2806621.052706] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2806621.052722] [] ? ptlrpc_check_set+0xff9/0x1bf0 [ptlrpc] <4>[2806621.052723] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2806621.052724] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2806621.052726] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[2806621.052727] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2806621.052728] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2806621.052746] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2806621.052747] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2806621.052749] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2806621.052766] [] ? ptlrpcd+0x20b/0x370 [ptlrpc] <4>[2806621.052767] Node 0 Normal per-cpu: <4>[2806621.052770] [] ? default_wake_function+0x0/0x20 <4>[2806621.052770] CPU 0: hi: 186, btch: 31 usd: 159 <4>[2806621.052771] CPU 1: hi: 186, btch: 31 usd: 31 <4>[2806621.052789] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2806621.052790] CPU 2: hi: 186, btch: 31 usd: 56 <4>[2806621.052791] CPU 3: hi: 186, btch: 31 usd: 30 <4>[2806621.052793] [] ? kthread+0x9e/0xc0 <4>[2806621.052794] CPU 4: hi: 186, btch: 31 usd: 20 <4>[2806621.052795] CPU 5: hi: 186, btch: 31 usd: 59 <4>[2806621.052797] [] ? child_rip+0xa/0x20 <4>[2806621.052798] CPU 6: hi: 186, btch: 31 usd: 2 <4>[2806621.052799] CPU 7: hi: 186, btch: 31 usd: 32 <4>[2806621.052801] [] ? kthread+0x0/0xc0 <4>[2806621.052803] active_anon:4843401 inactive_anon:166075 isolated_anon:0 <4>[2806621.052803] active_file:10972834 inactive_file:12862021 isolated_file:0 <4>[2806621.052804] unevictable:8448 dirty:1384 writeback:0 unstable:0 <4>[2806621.052804] free:914117 slab_reclaimable:22149918 slab_unreclaimable:13287514 <4>[2806621.052805] mapped:8494 shmem:222166 pagetables:10678 bounce:0 <4>[2806621.052805] Node 0 [] ? child_rip+0x0/0x20 <4>[2806621.052809] DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <6>[2806621.052810] Mem-Info: <4>[2806621.052811] lowmem_reserve[]:Node 0 0DMA per-cpu: <4>[2806621.052813] 1880CPU 0: hi: 0, btch: 1 usd: 0 <4>[2806621.052814] 258420CPU 1: hi: 0, btch: 1 usd: 0 <4>[2806621.052815] 258420CPU 2: hi: 0, btch: 1 usd: 0 <4>[2806621.052816] <4>[2806621.052817] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2806621.052818] Node 0 CPU 4: hi: 0, btch: 1 usd: 0 <4>[2806621.052820] DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2806621.052822] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2806621.052823] lowmem_reserve[]:CPU 6: hi: 0, btch: 1 usd: 0 <4>[2806621.052825] 0CPU 7: hi: 0, btch: 1 usd: 0 <4>[2806621.052826] 0Node 0 256540DMA32 per-cpu: <4>[2806621.052827] 256540CPU 0: hi: 186, btch: 31 usd: 0 <4>[2806621.052829] <4>[2806621.052829] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2806621.052830] Node 0 CPU 2: hi: 186, btch: 31 usd: 0 <4>[2806621.052833] Normal free:3255612kB min:67088kB low:83860kB high:100632kB active_anon:19373604kB inactive_anon:664300kB active_file:43891336kB inactive_file:51448084kB unevictable:33792kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:5536kB writeback:0kB mapped:33976kB shmem:888664kB slab_reclaimable:88599672kB slab_unreclaimable:53150056kB kernel_stack:74352kB pagetables:42712kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2806621.052836] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2806621.052836] lowmem_reserve[]:CPU 4: hi: 186, btch: 31 usd: 0 <4>[2806621.052838] 0CPU 5: hi: 186, btch: 31 usd: 0 <4>[2806621.052839] 0CPU 6: hi: 186, btch: 31 usd: 0 <4>[2806621.052840] 0CPU 7: hi: 186, btch: 31 usd: 0 <4>[2806621.052841] 0Node 0 <4>[2806621.052842] Normal per-cpu: <4>[2806621.052843] Node 0 CPU 0: hi: 186, btch: 31 usd: 156 <4>[2806621.052844] DMA: CPU 1: hi: 186, btch: 31 usd: 31 <4>[2806621.052845] 1*4kB CPU 2: hi: 186, btch: 31 usd: 56 <4>[2806621.052847] 1*8kB CPU 3: hi: 186, btch: 31 usd: 30 <4>[2806621.052848] 1*16kB CPU 4: hi: 186, btch: 31 usd: 20 <4>[2806621.052849] 1*32kB CPU 5: hi: 186, btch: 31 usd: 59 <4>[2806621.052851] 1*64kB CPU 6: hi: 186, btch: 31 usd: 1 <4>[2806621.052852] 0*128kB CPU 7: hi: 186, btch: 31 usd: 32 <4>[2806621.052853] 1*256kB active_anon:4843401 inactive_anon:166075 isolated_anon:0 <4>[2806621.052856] active_file:10972834 inactive_file:12862021 isolated_file:0 <4>[2806621.052856] unevictable:8448 dirty:1384 writeback:0 unstable:0 <4>[2806621.052856] free:914117 slab_reclaimable:22149918 slab_unreclaimable:13287514 <4>[2806621.052857] mapped:8494 shmem:222166 pagetables:10678 bounce:0 <4>[2806621.052857] 0*512kB Node 0 1*1024kB DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2806621.052862] 1*2048kB lowmem_reserve[]:3*4096kB 0= 15740kB <4>[2806621.052864] 1880Node 0 258420DMA32: 2584209*4kB <4>[2806621.052867] 3*8kB Node 0 4*16kB DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2806621.052871] 7*32kB lowmem_reserve[]:6*64kB 05*128kB 07*256kB 25654010*512kB 2565408*1024kB <4>[2806621.052876] 4*2048kB Node 0 88*4096kB Normal free:3255612kB min:67088kB low:83860kB high:100632kB active_anon:19373604kB inactive_anon:664300kB active_file:43891336kB inactive_file:51448084kB unevictable:33792kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:5536kB writeback:0kB mapped:33976kB shmem:888664kB slab_reclaimable:88599672kB slab_unreclaimable:53150056kB kernel_stack:74352kB pagetables:42712kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2806621.052881] = 385116kB <4>[2806621.052881] lowmem_reserve[]:Node 0 0Normal: 0813539*4kB 00*8kB 00*16kB <4>[2806621.052885] 0*32kB Node 0 0*64kB DMA: 0*128kB 1*4kB 0*256kB 1*8kB 0*512kB 1*16kB 0*1024kB 1*32kB 0*2048kB 1*64kB 0*4096kB 0*128kB = 3254156kB <4>[2806621.052891] 1*256kB 24058188 total pagecache pages <4>[2806621.052893] 0*512kB 0 pages in swap cache <4>[2806621.052894] 1*1024kB Swap cache stats: add 0, delete 0, find 0/0 <4>[2806621.052895] 1*2048kB Free swap = 0kB <4>[2806621.052896] 3*4096kB Total swap = 0kB <4>[2806621.052897] = 15740kB <4>[2806621.052897] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2806621.052900] Node 0 Normal: 813539*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3254156kB <4>[2806621.052903] 24058188 total pagecache pages <4>[2806621.052903] 0 pages in swap cache <4>[2806621.052904] Swap cache stats: add 0, delete 0, find 0/0 <4>[2806621.052905] Free swap = 0kB <4>[2806621.052905] Total swap = 0kB <4>[2806621.053111] ptlrpcd_7: page allocation failure. order:1, mode:0x20 <4>[2806621.053112] Pid: 14895, comm: ptlrpcd_7 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2806621.053113] Call Trace: <4>[2806621.053116] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2806621.053121] [] ? kiblnd_send+0x2a3/0x9b0 [ko2iblnd] <4>[2806621.053122] [] ? kmem_getpages+0x62/0x170 <4>[2806621.053124] [] ? fallback_alloc+0x1ba/0x270 <4>[2806621.053125] [] ? cache_grow+0x2cf/0x320 <4>[2806621.053126] [] ? ____cache_alloc_node+0x99/0x160 <4>[2806621.053128] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806621.053130] [] ? __kmalloc+0x199/0x230 <4>[2806621.053131] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806621.053133] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2806621.053150] [] ? after_reply+0x32a/0xec0 [ptlrpc] <4>[2806621.053151] [] ? thread_return+0x4e/0x7d0 <4>[2806621.053168] [] ? ptlrpc_check_set+0xff9/0x1bf0 [ptlrpc] <4>[2806621.053169] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[2806621.053187] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2806621.053205] [] ? ptlrpcd+0x20b/0x370 [ptlrpc] <4>[2806621.053207] [] ? default_wake_function+0x0/0x20 <4>[2806621.053225] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2806621.053227] [] ? kthread+0x9e/0xc0 <4>[2806621.053228] [] ? child_rip+0xa/0x20 <4>[2806621.053230] [] ? kthread+0x0/0xc0 <4>[2806621.053231] [] ? child_rip+0x0/0x20 <6>[2806621.053232] Mem-Info: <4>[2806621.053232] Node 0 DMA per-cpu: <4>[2806621.053233] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2806621.053234] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2806621.053235] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2806621.053235] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2806621.053236] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2806621.053236] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2806621.053237] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2806621.053237] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2806621.053238] Node 0 DMA32 per-cpu: <4>[2806621.053239] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2806621.053240] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2806621.053240] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2806621.053241] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2806621.053241] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2806621.053242] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2806621.053243] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2806621.053243] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2806621.053244] Node 0 Normal per-cpu: <4>[2806621.053244] CPU 0: hi: 186, btch: 31 usd: 151 <4>[2806621.053245] CPU 1: hi: 186, btch: 31 usd: 31 <4>[2806621.053246] CPU 2: hi: 186, btch: 31 usd: 56 <4>[2806621.053246] CPU 3: hi: 186, btch: 31 usd: 30 <4>[2806621.053247] CPU 4: hi: 186, btch: 31 usd: 20 <4>[2806621.053247] CPU 5: hi: 186, btch: 31 usd: 59 <4>[2806621.053248] CPU 6: hi: 186, btch: 31 usd: 1 <4>[2806621.053249] CPU 7: hi: 186, btch: 31 usd: 32 <4>[2806621.053251] active_anon:4843401 inactive_anon:166075 isolated_anon:0 <4>[2806621.053251] active_file:10972834 inactive_file:12862021 isolated_file:0 <4>[2806621.053252] unevictable:8448 dirty:1384 writeback:0 unstable:0 <4>[2806621.053252] free:914117 slab_reclaimable:22149918 slab_unreclaimable:13287514 <4>[2806621.053252] mapped:8494 shmem:222166 pagetables:10678 bounce:0 <4>[2806621.053253] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2806621.053256] lowmem_reserve[]: 0 1880 258420 258420 <4>[2806621.053257] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2806621.053261] lowmem_reserve[]: 0 0 256540 256540 <4>[2806621.053262] Node 0 Normal free:3255612kB min:67088kB low:83860kB high:100632kB active_anon:19373604kB inactive_anon:664300kB active_file:43891336kB inactive_file:51448084kB unevictable:33792kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:5536kB writeback:0kB mapped:33976kB shmem:888664kB slab_reclaimable:88599672kB slab_unreclaimable:53150056kB kernel_stack:74352kB pagetables:42712kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2806621.053265] lowmem_reserve[]: 0 0 0 0 <4>[2806621.053266] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2806621.053269] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2806621.053272] Node 0 Normal: 813539*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3254156kB <4>[2806621.053275] 24058188 total pagecache pages <4>[2806621.053276] 0 pages in swap cache <4>[2806621.053276] Swap cache stats: add 0, delete 0, find 0/0 <4>[2806621.053277] Free swap = 0kB <4>[2806621.053277] Total swap = 0kB <6>[2806621.389921] 67108863 pages RAM <6>[2806621.389922] 1015653 pages reserved <6>[2806621.389922] 18983753 pages shared <6>[2806621.389923] 45968924 pages non-shared <3>[2806621.389937] LustreError: 14891:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2806621.389939] LustreError: 14891:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1479647208 total bytes allocated by lnet <6>[2806621.718950] 67108863 pages RAM <6>[2806621.718951] 1015653 pages reserved <6>[2806621.718951] 18983691 pages shared <6>[2806621.718952] 45968700 pages non-shared <6>[2806622.047880] 67108863 pages RAM <6>[2806622.047881] 1015653 pages reserved <6>[2806622.047882] 18986251 pages shared <6>[2806622.047882] 45968655 pages non-shared <3>[2806622.047940] LustreError: 14895:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2806622.047943] LustreError: 14895:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <3>[2806622.047944] LustreError: 14895:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1479831336 total bytes allocated by lnet <3>[2806622.047945] LustreError: 14895:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) Skipped 1 previous similar message <4>[2806622.979023] Pid: 14890, comm: ptlrpcd_2 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2806622.988623] Call Trace: <4>[2806622.991786] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2806622.999498] [] ? kiblnd_send+0x2a3/0x9b0 [ko2iblnd] <4>[2806623.007213] [] ? kmem_getpages+0x62/0x170 <4>[2806623.013955] [] ? fallback_alloc+0x1ba/0x270 <4>[2806623.020918] [] ? cache_grow+0x2cf/0x320 <4>[2806623.027474] [] ? ____cache_alloc_node+0x99/0x160 <4>[2806623.034918] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806623.043548] [] ? __kmalloc+0x199/0x230 <4>[2806623.050009] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2806623.058640] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2806623.066766] [] ? after_reply+0x32a/0xec0 [ptlrpc] <4>[2806623.074302] [] ? ptlrpc_check_set+0xff9/0x1bf0 [ptlrpc] <4>[2806623.082402] [] ? try_to_del_timer_sync+0x7b/0xe0 <4>[2806623.089840] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2806623.097580] [] ? ptlrpcd+0x1db/0x370 [ptlrpc] <4>[2806623.104711] [] ? default_wake_function+0x0/0x20 <4>[2806623.112056] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2806623.119038] [] ? kthread+0x9e/0xc0 <4>[2806623.125116] [] ? child_rip+0xa/0x20 <4>[2806623.131285] [] ? kthread+0x0/0xc0 <4>[2806623.137274] [] ? child_rip+0x0/0x20 <6>[2806623.143436] Mem-Info: <4>[2806623.146409] Node 0 DMA per-cpu: <4>[2806623.150347] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2806623.156129] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2806623.161906] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2806623.167699] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2806623.173458] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2806623.179226] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2806623.184993] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2806623.190784] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2806623.196554] Node 0 DMA32 per-cpu: <4>[2806623.200701] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2806623.206479] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2806623.212248] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2806623.218037] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2806623.223840] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2806623.229613] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2806623.235392] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2806623.241153] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2806623.246923] Node 0 Normal per-cpu: <4>[2806623.251161] CPU 0: hi: 186, btch: 31 usd: 155 <4>[2806623.256929] CPU 1: hi: 186, btch: 31 usd: 31 <4>[2806623.262713] CPU 2: hi: 186, btch: 31 usd: 58 <4>[2806623.268494] CPU 3: hi: 186, btch: 31 usd: 21 <4>[2806623.274288] CPU 4: hi: 186, btch: 31 usd: 27 <4>[2806623.280057] CPU 5: hi: 186, btch: 31 usd: 72 <4>[2806623.285850] CPU 6: hi: 186, btch: 31 usd: 179 <4>[2806623.291628] CPU 7: hi: 186, btch: 31 usd: 109 <4>[2806623.297397] active_anon:4843401 inactive_anon:166075 isolated_anon:0 <4>[2806623.297398] active_file:10972845 inactive_file:12859131 isolated_file:0 <4>[2806623.297398] unevictable:8448 dirty:1268 writeback:0 unstable:0 <4>[2806623.297399] free:920029 slab_reclaimable:22148630 slab_unreclaimable:13285598 <4>[2806623.297399] mapped:8494 shmem:222166 pagetables:10678 bounce:0 <4>[2806623.335658] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2806623.377822] lowmem_reserve[]: 0 1880 258420 258420 <4>[2806623.383661] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2806623.426811] lowmem_reserve[]: 0 0 256540 256540 <4>[2806623.432393] Node 0 Normal free:3279260kB min:67088kB low:83860kB high:100632kB active_anon:19373604kB inactive_anon:664300kB active_file:43891380kB inactive_file:51436524kB unevictable:33792kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:5072kB writeback:0kB mapped:33976kB shmem:888664kB slab_reclaimable:88594520kB slab_unreclaimable:53142392kB kernel_stack:74352kB pagetables:42712kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2806623.488403] lowmem_reserve[]: 0 0 0 0 <4>[2806623.492998] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2806623.505720] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2806623.518847] Node 0 Normal: 815849*4kB 72*8kB 24*16kB 24*32kB 17*64kB 4*128kB 48*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3279012kB <4>[2806623.533011] 24055253 total pagecache pages <4>[2806623.538037] 0 pages in swap cache <4>[2806623.542166] Swap cache stats: add 0, delete 0, find 0/0 <4>[2806623.548420] Free swap = 0kB <4>[2806623.552052] Total swap = 0kB <6>[2806623.888237] 67108863 pages RAM <6>[2806623.892064] 1015653 pages reserved <6>[2806623.896304] 18986244 pages shared <6>[2806623.900426] 45968520 pages non-shared <3>[2806623.905226] LustreError: 14890:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2806623.929994] LustreError: 14890:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1479810504 total bytes allocated by lnet <4>[2807371.052470] ptlrpcd_5: page allocation failure. order:1, mode:0x20 <4>[2807371.059797] Pid: 14893, comm: ptlrpcd_5 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[2807371.069387] Call Trace: <4>[2807371.072538] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[2807371.080262] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[2807371.088870] [] ? kmem_getpages+0x62/0x170 <4>[2807371.095618] [] ? fallback_alloc+0x1ba/0x270 <4>[2807371.102553] [] ? cache_grow+0x2cf/0x320 <4>[2807371.109097] [] ? ____cache_alloc_node+0x99/0x160 <4>[2807371.116520] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2807371.125144] [] ? __kmalloc+0x199/0x230 <4>[2807371.131599] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[2807371.140209] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[2807371.148347] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[2807371.155980] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[2807371.164103] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[2807371.172631] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[2807371.180756] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[2807371.188487] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[2807371.195621] [] ? default_wake_function+0x0/0x20 <4>[2807371.202968] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[2807371.209907] [] ? kthread+0x9e/0xc0 <4>[2807371.215971] [] ? child_rip+0xa/0x20 <4>[2807371.222132] [] ? kthread+0x0/0xc0 <4>[2807371.228189] [] ? child_rip+0x0/0x20 <6>[2807371.234347] Mem-Info: <4>[2807371.237300] Node 0 DMA per-cpu: <4>[2807371.241235] CPU 0: hi: 0, btch: 1 usd: 0 <4>[2807371.247006] CPU 1: hi: 0, btch: 1 usd: 0 <4>[2807371.252775] CPU 2: hi: 0, btch: 1 usd: 0 <4>[2807371.258544] CPU 3: hi: 0, btch: 1 usd: 0 <4>[2807371.264314] CPU 4: hi: 0, btch: 1 usd: 0 <4>[2807371.270084] CPU 5: hi: 0, btch: 1 usd: 0 <4>[2807371.275853] CPU 6: hi: 0, btch: 1 usd: 0 <4>[2807371.281622] CPU 7: hi: 0, btch: 1 usd: 0 <4>[2807371.287395] Node 0 DMA32 per-cpu: <4>[2807371.291524] CPU 0: hi: 186, btch: 31 usd: 0 <4>[2807371.297294] CPU 1: hi: 186, btch: 31 usd: 0 <4>[2807371.303064] CPU 2: hi: 186, btch: 31 usd: 0 <4>[2807371.308834] CPU 3: hi: 186, btch: 31 usd: 0 <4>[2807371.314603] CPU 4: hi: 186, btch: 31 usd: 0 <4>[2807371.320376] CPU 5: hi: 186, btch: 31 usd: 0 <4>[2807371.326145] CPU 6: hi: 186, btch: 31 usd: 0 <4>[2807371.331914] CPU 7: hi: 186, btch: 31 usd: 0 <4>[2807371.337684] Node 0 Normal per-cpu: <4>[2807371.341911] CPU 0: hi: 186, btch: 31 usd: 168 <4>[2807371.347682] CPU 1: hi: 186, btch: 31 usd: 64 <4>[2807371.353451] CPU 2: hi: 186, btch: 31 usd: 74 <4>[2807371.359223] CPU 3: hi: 186, btch: 31 usd: 32 <4>[2807371.364994] CPU 4: hi: 186, btch: 31 usd: 59 <4>[2807371.370761] CPU 5: hi: 186, btch: 31 usd: 169 <4>[2807371.376530] CPU 6: hi: 186, btch: 31 usd: 24 <4>[2807371.382300] CPU 7: hi: 186, btch: 31 usd: 69 <4>[2807371.388071] active_anon:4843485 inactive_anon:166103 isolated_anon:0 <4>[2807371.388071] active_file:10965270 inactive_file:16533305 isolated_file:0 <4>[2807371.388072] unevictable:8448 dirty:447 writeback:657 unstable:0 <4>[2807371.388072] free:1772984 slab_reclaimable:18830427 slab_unreclaimable:12083446 <4>[2807371.388072] mapped:8490 shmem:222298 pagetables:10678 bounce:0 <4>[2807371.426453] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2807371.468495] lowmem_reserve[]: 0 1880 258420 258420 <4>[2807371.474323] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[2807371.517434] lowmem_reserve[]: 0 0 256540 256540 <4>[2807371.522982] Node 0 Normal free:6680664kB min:67088kB low:83860kB high:100632kB active_anon:19373940kB inactive_anon:664412kB active_file:43861080kB inactive_file:66141368kB unevictable:33792kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:1788kB writeback:4368kB mapped:33960kB shmem:889192kB slab_reclaimable:75321708kB slab_unreclaimable:48335724kB kernel_stack:74352kB pagetables:42712kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[2807371.573794] lowmem_reserve[]: 0 0 0 0 <4>[2807371.578368] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[2807371.591026] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[2807371.604192] Node 0 Normal: 1673493*4kB 1642*8kB 459*16kB 230*32kB 43*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6725076kB <4>[2807371.618679] 27717066 total pagecache pages <4>[2807371.623674] 0 pages in swap cache <4>[2807371.627797] Swap cache stats: add 0, delete 0, find 0/0 <4>[2807371.634051] Free swap = 0kB <4>[2807371.637686] Total swap = 0kB <6>[2807372.010678] 67108863 pages RAM <6>[2807372.014505] 1015653 pages reserved <6>[2807372.018720] 22702702 pages shared <6>[2807372.022836] 41348143 pages non-shared <3>[2807372.027400] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[2807372.052135] LustreError: 14893:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1483664976 total bytes allocated by lnet <4>[2807651.795570] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2807655.707723] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2807658.094941] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2808291.077819] Lustre: Unmounted atlas1-client <4>[2822040.425667] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2822044.086384] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[2823708.132279] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 11978@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8807292a5500/0xe00c22b32c372c39 lrc: 4/0,0 mode: PR/PR res: [0x2002cb91b:0xb60:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 11978@gni100 remote: 0xc8ff233d99939d1b expref: 14 pid: 16075 timeout: 7118376573 lvb_type: 0 <3>[2823708.175520] LustreError: 15349:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 11978@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8807292a5500/0xe00c22b32c372c39 lrc: 2/0,0 mode: --/PR res: [0x2002cb91b:0xb60:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x44a01000000020 nid: 11978@gni100 remote: 0xc8ff233d99939d1b expref: 4 pid: 16075 timeout: 7118376573 lvb_type: 0 <3>[2823714.132247] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9063@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff88079565b740/0xe00c22b32c44c14f lrc: 4/0,0 mode: PR/PR res: [0x2002c6858:0xb3:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x60200000000020 nid: 9063@gni100 remote: 0x8a1a55a983d28fc9 expref: 9 pid: 15519 timeout: 7118382423 lvb_type: 0 <3>[2823714.175043] LustreError: 15762:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9063@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff88079565b740/0xe00c22b32c44c14f lrc: 4/0,0 mode: PR/PR res: [0x2002c6858:0xb3:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x60200000000020 nid: 9063@gni100 remote: 0x8a1a55a983d28fc9 expref: 7 pid: 15519 timeout: 7118382423 lvb_type: 0 <4>[2836440.354399] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2836444.195435] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2836446.239715] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2843109.833846] Lustre: atlas1-MDT0000: Client 74b7233f-1871-02ea-fa7d-7afdd823b5e8 (at 10.36.205.206@o2ib) reconnecting <4>[2872849.820954] Lustre: atlas1-MDT0000: haven't heard from client 2fac5f95-d2e0-060a-5a7c-31616eb6471c (at 10263@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c916800, cur 1459145932 expire 1459145032 last 1459144580 <4>[2872849.846396] Lustre: Skipped 6 previous similar messages <6>[2891520.786834] Lustre: atlas1-MDT0000-mdc-ffff883f78225c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2891520.786920] Lustre: client wants to enable acl, but mdt not! <6>[2891520.786922] Lustre: Layout lock feature supported. <6>[2891520.810583] Lustre: Skipped 1008 previous similar messages <4>[2891520.928797] Lustre: Mounted atlas1-client <4>[2894043.451635] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2894047.462956] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2894634.514989] Lustre: Unmounted atlas1-client <4>[2908443.543674] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2908447.273545] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2922838.406323] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2922842.101732] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2929694.767447] Lustre: atlas1-MDT0000: haven't heard from client 94e18009-702d-99ee-7036-8cf7a79b04cc (at 16251@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f792c9400, cur 1459202777 expire 1459201877 last 1459201425 <4>[2965794.733239] Lustre: atlas1-MDT0000: haven't heard from client 7f14dd1d-0446-1f44-74be-ed04d8047264 (at 714@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e9fd000, cur 1459238877 expire 1459237977 last 1459237525 <6>[2977920.686595] Lustre: atlas1-MDT0000-mdc-ffff883f309da000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[2977920.686735] Lustre: client wants to enable acl, but mdt not! <6>[2977920.686737] Lustre: Layout lock feature supported. <6>[2977920.710342] Lustre: Skipped 1008 previous similar messages <4>[2977920.829275] Lustre: Mounted atlas1-client <4>[2980445.511748] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2980449.143646] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2980994.262186] Lustre: Unmounted atlas1-client <4>[2981994.718110] Lustre: atlas1-MDT0000: haven't heard from client 0dd788de-5b0f-793d-08ea-fc7cfa0ae0bc (at 14404@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7dfeac00, cur 1459255077 expire 1459254177 last 1459253725 <4>[2984738.715500] Lustre: atlas1-MDT0000: haven't heard from client b5dbde3a-cd49-f09c-4b0d-a0d04fef92d9 (at 10.38.144.7@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4232e400, cur 1459257821 expire 1459256921 last 1459256469 <4>[2984738.741386] Lustre: Skipped 2 previous similar messages <4>[2986805.713601] Lustre: atlas1-MDT0000: haven't heard from client c59c82fb-a924-2405-8946-f513b8b57fcc (at 10.38.145.112@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f35600400, cur 1459259888 expire 1459258988 last 1459258536 <4>[2987256.713032] Lustre: atlas1-MDT0000: haven't heard from client f770895b-50e0-044f-7a5d-cf39474c345f (at 478@gni3) in 913 seconds. I think it's dead, and I am evicting it. exp ffff883f392cc000, cur 1459260339 expire 1459259439 last 1459259426 <4>[2990730.709783] Lustre: atlas1-MDT0000: haven't heard from client 757fc096-380e-162e-e69e-cb145b69a123 (at 10.36.247.149@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d431c00, cur 1459263813 expire 1459262913 last 1459262461 <4>[2991181.710811] Lustre: atlas1-MDT0000: haven't heard from client 7c9e01b2-1a19-cca7-06a6-c7ad658ae5b3 (at 15538@gni100) in 1256 seconds. I think it's dead, and I am evicting it. exp ffff883f7abcb800, cur 1459264264 expire 1459263364 last 1459263008 <4>[2991181.736187] Lustre: Skipped 38 previous similar messages <4>[2994838.828481] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2994842.550979] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[2994844.575337] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3001027.700556] Lustre: atlas1-MDT0000: haven't heard from client a689d37c-2609-611d-38ad-0cb63c392922 (at 567@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f37917c00, cur 1459274110 expire 1459273210 last 1459272758 <4>[3001027.725573] Lustre: Skipped 2 previous similar messages <4>[3009246.429688] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3009250.107721] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3012324.691360] Lustre: atlas1-MDT0000: haven't heard from client 7820f0b3-7945-884c-0839-a27822306106 (at 11436@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ee06800, cur 1459285407 expire 1459284507 last 1459284055 <4>[3012324.716746] Lustre: Skipped 11 previous similar messages <3>[3015266.950460] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9306@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff88349167f7c0/0xe00c22a8d4236c74 lrc: 4/0,0 mode: PR/PR res: [0x2002c5910:0xa8c:0x0].0 bits 0x1b rrc: 2 type: IBT flags: 0x60200000000020 nid: 9306@gni100 remote: 0x8aeb80d79621c77e expref: 307806 pid: 15521 timeout: 7309935142 lvb_type: 0 <3>[3015266.993880] LustreError: 15628:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9306@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8830f621f780/0xe00c2276a4f99079 lrc: 4/0,0 mode: PR/PR res: [0x2002c9103:0x5d11:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x60200000000020 nid: 9306@gni100 remote: 0x8aeb80d79491bc0b expref: 307782 pid: 15537 timeout: 7310216002 lvb_type: 0 <3>[3015267.035826] LustreError: 15628:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 1 previous similar message <3>[3017267.948762] LustreError: 4:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 5692@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8802908653c0/0xe00c22b61aa6d376 lrc: 4/0,0 mode: PR/PR res: [0x20029be7f:0x172de:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x60200000000020 nid: 5692@gni100 remote: 0x73d294e8c747c92f expref: 11 pid: 16212 timeout: 7311936163 lvb_type: 0 <3>[3017267.992003] LustreError: 15688:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 5692@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8802908653c0/0xe00c22b61aa6d376 lrc: 2/0,0 mode: --/PR res: [0x20029be7f:0x172de:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x44a01000000020 nid: 5692@gni100 remote: 0x73d294e8c747c92f expref: 4 pid: 16212 timeout: 7311936163 lvb_type: 0 <3>[3017268.033620] LustreError: 15688:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 1 previous similar message <4>[3025820.680970] Lustre: atlas1-MDT0000: haven't heard from client 83b4ac4d-18e7-d1a7-5d65-8984a4294fa0 (at 7528@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a91d800, cur 1459298903 expire 1459298003 last 1459297551 <4>[3027789.012960] Lustre: atlas1-MDT0000: Client 7740451a-a018-d028-6b8e-007b01e2912b (at 1128@gni100) reconnecting <4>[3027799.644414] Lustre: atlas1-MDT0000: Client 4f12890b-ffc1-322c-8143-65c6406b1b45 (at 5292@gni100) reconnecting <4>[3034362.673014] Lustre: atlas1-MDT0000: Client d36cb535-fabf-e1f1-38f3-9099d31feb0e (at 13408@gni100) reconnecting <4>[3036462.104160] Lustre: atlas1-MDT0000: Client c81dae8a-2cbd-c0bc-c16f-42e945a79583 (at 4552@gni100) reconnecting <4>[3037455.107968] Lustre: atlas1-MDT0000: Client 97cb53bb-ed04-a027-6124-1ba5b203befe (at 16293@gni100) reconnecting <4>[3037779.111676] Lustre: atlas1-MDT0000: Client dc4ec6aa-bc85-124f-072b-ece6c2e7816d (at 1132@gni100) reconnecting <4>[3038907.665864] Lustre: atlas1-MDT0000: haven't heard from client 92e0db3e-140a-3e81-f34f-70cf4976b946 (at 15488@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ce96000, cur 1459311990 expire 1459311090 last 1459310638 <4>[3041163.662100] Lustre: atlas1-MDT0000: haven't heard from client 597ec4ec-44f0-5b9c-0f90-f928d00b1c2f (at 12315@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d0b9400, cur 1459314246 expire 1459313346 last 1459312894 <4>[3049248.656862] Lustre: atlas1-MDT0000: haven't heard from client 6d7a9036-828c-8627-534d-29c433e067eb (at 11@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d2dac00, cur 1459322331 expire 1459321431 last 1459320979 <4>[3053152.653384] Lustre: atlas1-MDT0000: haven't heard from client 22f4624f-5417-0b63-83d7-75a856108c82 (at 15334@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e9f7000, cur 1459326235 expire 1459325335 last 1459324883 <6>[3064321.185900] Lustre: atlas1-MDT0000-mdc-ffff883f3f09cc00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3064321.186089] Lustre: client wants to enable acl, but mdt not! <6>[3064321.186090] Lustre: Layout lock feature supported. <6>[3064321.209666] Lustre: Skipped 1008 previous similar messages <4>[3064321.325776] Lustre: Mounted atlas1-client <4>[3066868.104390] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3066871.853500] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3066873.907766] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3067224.820204] Lustre: Unmounted atlas1-client <4>[3074121.629517] Lustre: atlas1-MDT0000: haven't heard from client f3b0d6fd-7f4c-0bc7-dbc7-82e49cac4c7e (at 10.36.247.137@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3f4fb000, cur 1459347204 expire 1459346304 last 1459345852 <4>[3078917.624286] Lustre: atlas1-MDT0000: haven't heard from client 2be76239-f7cd-72b0-4697-3e87e7fe8c0f (at 10.38.144.58@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f32c4e400, cur 1459352000 expire 1459351100 last 1459350648 <4>[3081242.981416] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3081246.616286] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3090718.614486] Lustre: atlas1-MDT0000: haven't heard from client 723a7cb2-c7d9-b11f-76c1-8971ca546539 (at 10.36.202.184@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f42cad000, cur 1459363801 expire 1459362901 last 1459362449 <4>[3091169.615629] Lustre: atlas1-MDT0000: haven't heard from client 7b62e6dd-d3af-e23f-8cc2-ff3a437be50a (at 5255@gni100) in 998 seconds. I think it's dead, and I am evicting it. exp ffff883f4822a400, cur 1459364252 expire 1459363352 last 1459363254 <4>[3091169.640821] Lustre: Skipped 7 previous similar messages <4>[3092600.614029] Lustre: atlas1-MDT0000: haven't heard from client 916bc205-d2af-7e4f-d687-b87db30bb667 (at 10069@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3d9a8800, cur 1459365683 expire 1459364783 last 1459364331 <4>[3092600.639422] Lustre: Skipped 2 previous similar messages <4>[3092895.540228] Lustre: atlas1-MDT0000: Client ea6993f1-4daf-f76c-6af3-77aa2a7195c1 (at 10069@gni100) reconnecting <4>[3095638.261346] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3095641.906112] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3097725.609545] Lustre: atlas1-MDT0000: haven't heard from client e6b740ad-9cac-db5c-e08b-3dc071c368b1 (at 17069@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a388400, cur 1459370808 expire 1459369908 last 1459369456 <4>[3097725.634933] Lustre: Skipped 2 previous similar messages <4>[3101600.606064] Lustre: atlas1-MDT0000: haven't heard from client 290df42b-ac16-92fb-3702-488a200b1808 (at 4568@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880df0f60000, cur 1459374683 expire 1459373783 last 1459373331 <4>[3101600.631360] Lustre: Skipped 1 previous similar message <4>[3107557.955080] Lustre: atlas1-MDT0000: Client 25c8ad6c-3c52-cf8e-2054-dca2a846806a (at 14664@gni100) reconnecting <4>[3114740.594691] Lustre: atlas1-MDT0000: haven't heard from client 581047c4-9eb9-f878-a1d9-f2d078dc89b6 (at 10350@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d1b6800, cur 1459387823 expire 1459386923 last 1459386471 <4>[3138938.571047] Lustre: atlas1-MDT0000: haven't heard from client c2416dc2-874c-4ed9-8e90-0e9959cf498b (at 7722@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e8a1c00, cur 1459412021 expire 1459411121 last 1459410669 <3>[3142231.830198] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 18257@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883ae1abfd00/0xe00c22baea5e47c7 lrc: 4/0,0 mode: PR/PR res: [0x2002c6858:0xb3:0x0].0 bits 0x13 rrc: 10 type: IBT flags: 0x60200000000020 nid: 18257@gni100 remote: 0x8b0b99839519b4e9 expref: 22 pid: 15767 timeout: 7436900912 lvb_type: 0 <3>[3142231.873430] LustreError: 15327:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 18257@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883ae1abfd00/0xe00c22baea5e47c7 lrc: 4/0,0 mode: PR/PR res: [0x2002c6858:0xb3:0x0].0 bits 0x13 rrc: 10 type: IBT flags: 0x60200000000020 nid: 18257@gni100 remote: 0x8b0b99839519b4e9 expref: 19 pid: 15767 timeout: 7436900912 lvb_type: 0 <4>[3142550.567154] Lustre: atlas1-MDT0000: haven't heard from client feb7d116-4e32-a432-df60-ed03bb6bccb2 (at 14687@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7cdbbc00, cur 1459415633 expire 1459414733 last 1459414281 <4>[3146906.563731] Lustre: atlas1-MDT0000: haven't heard from client 3ab82e42-e254-7151-06f8-5326aa5b1d74 (at 7631@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7a7a4000, cur 1459419989 expire 1459419089 last 1459418637 <6>[3150720.938736] Lustre: atlas1-MDT0000-mdc-ffff883f45c4e000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3150720.938884] Lustre: client wants to enable acl, but mdt not! <6>[3150720.938885] Lustre: Layout lock feature supported. <6>[3150720.962499] Lustre: Skipped 1008 previous similar messages <4>[3150721.080074] Lustre: Mounted atlas1-client <4>[3153241.597772] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3153245.451981] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3153247.464991] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3153622.352206] Lustre: Unmounted atlas1-client <4>[3154384.555650] Lustre: atlas1-MDT0000: haven't heard from client 0a503fb6-17cb-1def-e568-55dc5fc057d7 (at 3920@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e184c00, cur 1459427467 expire 1459426567 last 1459426115 <4>[3154835.555253] Lustre: atlas1-MDT0000: haven't heard from client 5bbd3061-dfbc-d9fc-ecfd-267a153c03c3 (at 10.36.202.181@o2ib) in 976 seconds. I think it's dead, and I am evicting it. exp ffff883f3570fc00, cur 1459427918 expire 1459427018 last 1459426942 <4>[3167530.542456] Lustre: atlas1-MDT0000: haven't heard from client 25082499-b82a-c4b3-1b17-d13e8dfd7d01 (at 8077@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c7c4c00, cur 1459440613 expire 1459439713 last 1459439261 <4>[3167530.567741] Lustre: Skipped 7 previous similar messages <4>[3167641.860015] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3167645.553123] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3167981.542017] Lustre: atlas1-MDT0000: haven't heard from client baf2041b-b3a9-7723-4f90-b2e618ee7da4 (at 10.36.207.134@o2ib) in 1000 seconds. I think it's dead, and I am evicting it. exp ffff883f4846c400, cur 1459441064 expire 1459440164 last 1459440064 <4>[3167981.567969] Lustre: Skipped 10 previous similar messages <4>[3172313.537567] Lustre: atlas1-MDT0000: haven't heard from client 50aa7826-e83b-3022-fb1e-cd8c1c2fbacd (at 5366@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7bae7000, cur 1459445396 expire 1459444496 last 1459444044 <4>[3180755.528920] Lustre: atlas1-MDT0000: haven't heard from client 05a92b79-dc8e-74d7-1098-9038162ce76e (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f47e1ec00, cur 1459453838 expire 1459452938 last 1459452486 <4>[3180755.554888] Lustre: Skipped 32 previous similar messages <4>[3210889.977047] Lustre: atlas1-MDT0000: Client 9f937eb7-3406-d115-0802-3f64a30d9a92 (at 7728@gni100) reconnecting <4>[3220570.488074] Lustre: atlas1-MDT0000: haven't heard from client 4f6736a0-8641-47ee-3a84-9646fba47402 (at 14439@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7bc31c00, cur 1459493653 expire 1459492753 last 1459492301 <6>[3237120.955446] Lustre: atlas1-MDT0000-mdc-ffff883f56b85c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3237120.955603] Lustre: client wants to enable acl, but mdt not! <6>[3237120.955605] Lustre: Layout lock feature supported. <6>[3237120.979194] Lustre: Skipped 1008 previous similar messages <4>[3237121.109998] Lustre: Mounted atlas1-client <4>[3237683.470337] Lustre: atlas1-MDT0000: haven't heard from client 0963808e-cc80-b5da-6648-a001041de4e0 (at 4372@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e8e0800, cur 1459510766 expire 1459509866 last 1459509414 <4>[3239648.986678] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3239652.704008] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3240053.861496] Lustre: Unmounted atlas1-client <4>[3246168.461301] Lustre: atlas1-MDT0000: haven't heard from client 9848c88b-edde-1576-afc3-2ada51ebfc49 (at 11852@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ad90800, cur 1459519251 expire 1459518351 last 1459517899 <4>[3248861.458556] Lustre: atlas1-MDT0000: haven't heard from client 42c34f1d-0a12-2a2b-349e-c1594c9b9941 (at 31@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f49570000, cur 1459521944 expire 1459521044 last 1459520592 <4>[3254037.987564] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3254041.695766] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3257553.449747] Lustre: atlas1-MDT0000: haven't heard from client 24b0ca9d-1c46-d0ad-51ae-3ab0d8800b90 (at 8786@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7eddd000, cur 1459530636 expire 1459529736 last 1459529284 <4>[3261299.446065] Lustre: atlas1-MDT0000: haven't heard from client 1d2af85b-9da3-d07e-621e-dbf28049107d (at 10.36.202.185@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f39b11c00, cur 1459534382 expire 1459533482 last 1459533030 <4>[3268439.900246] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3268443.597066] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3288617.353787] Lustre: atlas1-MDT0000: Client f0a22616-ed06-dfa8-767a-d50e6352150f (at 6634@gni100) reconnecting <4>[3294662.414439] Lustre: atlas1-MDT0000: haven't heard from client d97737d0-69a9-8338-b37e-90ae6fde56b5 (at 7654@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c9d5400, cur 1459567745 expire 1459566845 last 1459566393 <4>[3294662.439718] Lustre: Skipped 7 previous similar messages <4>[3316328.220417] Lustre: atlas1-MDT0000: Client f53249cc-7885-3070-a3d1-27ed28e04b2b (at 12181@gni100) reconnecting <6>[3323520.285274] Lustre: atlas1-MDT0000-mdc-ffff883f32149c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3323520.285413] Lustre: client wants to enable acl, but mdt not! <6>[3323520.285414] Lustre: Layout lock feature supported. <6>[3323520.309029] Lustre: Skipped 1008 previous similar messages <4>[3323520.427398] Lustre: Mounted atlas1-client <4>[3326041.305345] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3326045.002656] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3326392.679878] Lustre: Unmounted atlas1-client <4>[3329792.415273] Lustre: atlas1-MDT0000: Client eee8d953-4fc4-8a29-4105-9d18c1a4c47d (at 7430@gni100) reconnecting <4>[3330661.379858] Lustre: atlas1-MDT0000: haven't heard from client cfc0a71d-4a5c-88f6-31a4-567ba02b080d (at 9168@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e298800, cur 1459603744 expire 1459602844 last 1459602392 <4>[3330661.405192] Lustre: Skipped 2 previous similar messages <4>[3332235.383117] Lustre: atlas1-MDT0000: Client 0fac1d5e-b2cc-6e0f-92c6-9a2678ed0aab (at 14741@gni100) reconnecting <4>[3336915.373623] Lustre: atlas1-MDT0000: haven't heard from client d0f5e5df-c003-20ac-6e8c-55a5a3ec6869 (at 15163@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f399ce400, cur 1459609998 expire 1459609098 last 1459608646 <4>[3338518.372899] Lustre: atlas1-MDT0000: haven't heard from client f52cabd4-378a-1824-f0d7-3f06852a0783 (at 10093@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f793c0000, cur 1459611601 expire 1459610701 last 1459610249 <4>[3340437.879246] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3340441.580044] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3344799.370123] Lustre: atlas1-MDT0000: haven't heard from client c9b6c538-a050-0faa-cb53-c35755a16a3b (at 14014@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7b8dd000, cur 1459617882 expire 1459616982 last 1459616530 <4>[3352443.360687] Lustre: atlas1-MDT0000: haven't heard from client 58cf5881-de32-cfac-0a4b-ec1c84928b57 (at 1807@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f78721800, cur 1459625526 expire 1459624626 last 1459624174 <4>[3354838.943358] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3354842.616497] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3357054.559647] Lustre: atlas1-MDT0000: Client 2cc18b00-6e8e-7eeb-e39f-5e8ea635345e (at 9024@gni100) reconnecting <4>[3373265.342529] Lustre: atlas1-MDT0000: haven't heard from client e244f9af-b5ed-6a3d-b87d-ab6bdf252b54 (at 18230@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7c4c3c00, cur 1459646348 expire 1459645448 last 1459644996 <4>[3374790.341060] Lustre: atlas1-MDT0000: haven't heard from client 8fb1d8ea-4686-36e2-cd4a-d0ec5b644206 (at 14721@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f781fc000, cur 1459647873 expire 1459646973 last 1459646521 <4>[3397403.320957] Lustre: atlas1-MDT0000: haven't heard from client aa08a47f-6aaa-8dc0-04d5-662f60283f19 (at 7099@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ab80800, cur 1459670486 expire 1459669586 last 1459669134 <6>[3409920.152691] Lustre: atlas1-MDT0000-mdc-ffff883f32fe3000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3409920.152840] Lustre: client wants to enable acl, but mdt not! <6>[3409920.152842] Lustre: Layout lock feature supported. <6>[3409920.176439] Lustre: Skipped 1008 previous similar messages <4>[3409920.293651] Lustre: Mounted atlas1-client <4>[3411219.308019] Lustre: atlas1-MDT0000: haven't heard from client 49042a98-ab10-0858-f26b-a04714610641 (at 14498@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e0a7000, cur 1459684302 expire 1459683402 last 1459682950 <4>[3411219.333397] Lustre: Skipped 1 previous similar message <4>[3412440.052104] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3412443.732369] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3412445.896351] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3412702.346199] Lustre: Unmounted atlas1-client <4>[3412821.754714] Lustre: atlas1-MDT0000: Client 5d6d506f-2672-765c-59ff-16b43afa29b7 (at 15339@gni100) reconnecting <4>[3426844.334554] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3426847.961211] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3428545.292573] Lustre: atlas1-MDT0000: haven't heard from client a5641c59-d7b0-9128-5d50-23b6f41b0b7c (at 8543@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7d54b000, cur 1459701628 expire 1459700728 last 1459700276 <4>[3428545.323120] Lustre: Skipped 1 previous similar message <4>[3430809.291570] Lustre: atlas1-MDT0000: haven't heard from client 13d6b1f2-b06a-fd1d-b102-c3543ba13e84 (at 10.36.247.147@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f43250c00, cur 1459703892 expire 1459702992 last 1459702540 <4>[3441237.887028] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3441241.544426] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[3447001.397594] LustreError: 138-a: atlas1-MDT0000: A client on nid 9310@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[3453909.271198] Lustre: atlas1-MDT0000: haven't heard from client 95da064b-fe40-fa71-d114-e33b8bd7bdec (at 13600@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7e23c000, cur 1459726992 expire 1459726092 last 1459725640 <4>[3457129.307095] Lustre: atlas1-MDT0000: Client 4dc1b692-3415-0d25-ab93-fece4537a5a0 (at 2256@gni100) reconnecting <4>[3457725.138448] Lustre: atlas1-MDT0000: Client 4dc1b692-3415-0d25-ab93-fece4537a5a0 (at 2256@gni100) reconnecting <4>[3472529.254191] Lustre: atlas1-MDT0000: haven't heard from client 50a4902f-bf4d-944b-df63-f9391c6fb012 (at 13054@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ba85400, cur 1459745612 expire 1459744712 last 1459744260 <4>[3483432.228215] Lustre: atlas1-MDT0000: Client fa05b95b-0169-715d-98f0-fe8e395e8871 (at 2251@gni100) reconnecting <4>[3485377.242156] Lustre: atlas1-MDT0000: haven't heard from client 55e135c3-5b76-f726-d712-eed0835252e6 (at 1206@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7dd3d800, cur 1459758460 expire 1459757560 last 1459757108 <4>[3488069.239331] Lustre: atlas1-MDT0000: haven't heard from client f7b47e1e-7399-2b62-a6a4-da7a062e87f3 (at 7663@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7aae5400, cur 1459761152 expire 1459760252 last 1459759800 <4>[3493091.234284] Lustre: atlas1-MDT0000: haven't heard from client 8eabd1c0-91dc-dbe6-1a25-ba5cde4e439f (at 5606@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f78455400, cur 1459766174 expire 1459765274 last 1459764822 <4>[3493542.235098] Lustre: atlas1-MDT0000: haven't heard from client 655002b7-87ce-3bf6-d55b-7459d000859a (at 26@gni3) in 1242 seconds. I think it's dead, and I am evicting it. exp ffff883f34bea400, cur 1459766625 expire 1459765725 last 1459765383 <6>[3496320.335229] Lustre: atlas1-MDT0000-mdc-ffff883fa4c22400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3496320.335423] Lustre: client wants to enable acl, but mdt not! <6>[3496320.335424] Lustre: Layout lock feature supported. <6>[3496320.359002] Lustre: Skipped 1008 previous similar messages <4>[3496320.477125] Lustre: Mounted atlas1-client <4>[3498841.855091] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3498845.584537] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3498942.934623] Lustre: Unmounted atlas1-client <4>[3511806.218671] Lustre: atlas1-MDT0000: haven't heard from client 167bb4dc-8102-c7b3-d7c0-8e2505f8ce7a (at 10.36.247.171@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3b063000, cur 1459784889 expire 1459783989 last 1459783537 <4>[3512257.217424] Lustre: atlas1-MDT0000: haven't heard from client ab654bbe-4e5b-761e-69c8-f5d2ed2c7ccc (at 9091@gni100) in 1175 seconds. I think it's dead, and I am evicting it. exp ffff883f795ab800, cur 1459785340 expire 1459784440 last 1459784165 <4>[3512751.804407] Lustre: atlas1-MDT0000: Client e397f791-ebb4-3697-aa5c-50dc6c46a47c (at 1085@gni100) reconnecting <4>[3513239.045617] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3513242.735075] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3513244.745034] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3515312.994838] Lustre: atlas1-MDT0000: Client 0e48431f-a1b5-ab45-b7b4-3dfeaba2686f (at 4564@gni100) reconnecting <4>[3516526.214823] Lustre: atlas1-MDT0000: haven't heard from client 10714163-c4d9-459b-ba7d-ef133f61f417 (at 30@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883fa8250400, cur 1459789609 expire 1459788709 last 1459788257 <3>[3520217.989152] LustreError: 138-a: atlas1-MDT0000: A client on nid 91@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[3521141.210722] Lustre: atlas1-MDT0000: haven't heard from client c7823d13-2063-4b33-d93a-a4df602afdd7 (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f33a6f000, cur 1459794224 expire 1459793324 last 1459792872 <4>[3521141.236719] Lustre: Skipped 81 previous similar messages <4>[3521592.210092] Lustre: atlas1-MDT0000: haven't heard from client c233b159-69fc-58e4-a86f-d3ecd19129df (at 12896@gni100) in 1318 seconds. I think it's dead, and I am evicting it. exp ffff883f47fd5000, cur 1459794675 expire 1459793775 last 1459793357 <4>[3522593.211643] Lustre: atlas1-MDT0000: haven't heard from client d5e5d434-0168-6d76-d1d2-203dd3b919de (at 11125@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883fd8c7cc00, cur 1459795676 expire 1459794776 last 1459794324 <4>[3524522.207932] Lustre: atlas1-MDT0000: haven't heard from client 617fd176-bd99-4e2a-2020-d9b8e627498d (at 4544@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f48394000, cur 1459797605 expire 1459796705 last 1459796253 <4>[3524522.233228] Lustre: Skipped 1 previous similar message <4>[3524973.206788] Lustre: atlas1-MDT0000: haven't heard from client 4deee134-b72f-5d45-a111-5594af2d5bb5 (at 11604@gni100) in 1171 seconds. I think it's dead, and I am evicting it. exp ffff883f47fde000, cur 1459798056 expire 1459797156 last 1459796885 <4>[3525424.206548] Lustre: atlas1-MDT0000: haven't heard from client 4c8de1f6-7bec-f414-28ca-83bb9cf9275e (at 9318@gni100) in 921 seconds. I think it's dead, and I am evicting it. exp ffff883f429b7400, cur 1459798507 expire 1459797607 last 1459797586 <4>[3525424.231725] Lustre: Skipped 2 previous similar messages <4>[3526576.205443] Lustre: atlas1-MDT0000: haven't heard from client 6a76073c-1320-6b33-e1af-dbe31ba6853f (at 11604@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3784d800, cur 1459799659 expire 1459798759 last 1459798307 <4>[3527644.160563] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3527647.824785] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3527899.203505] Lustre: atlas1-MDT0000: haven't heard from client 4c30cb78-7590-3344-080a-0acf1f2d1920 (at 17069@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f37b0ac00, cur 1459800982 expire 1459800082 last 1459799630 <4>[3527899.228901] Lustre: Skipped 1 previous similar message <4>[3528350.203131] Lustre: atlas1-MDT0000: haven't heard from client a24f1495-463f-214d-39e6-059fafa1c5b2 (at 91@gni100) in 980 seconds. I think it's dead, and I am evicting it. exp ffff883f339fcc00, cur 1459801433 expire 1459800533 last 1459800453 <4>[3528350.228184] Lustre: Skipped 1 previous similar message <4>[3529765.202900] Lustre: atlas1-MDT0000: haven't heard from client 4aaa3ad7-1924-5d3b-185b-14cdd06291d9 (at 91@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3d141c00, cur 1459802848 expire 1459801948 last 1459801496 <4>[3531081.202182] Lustre: atlas1-MDT0000: haven't heard from client ed52a46d-b6af-064b-8d2e-716a79aa3371 (at 91@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f348ce000, cur 1459804164 expire 1459803264 last 1459802812 <4>[3531532.205179] Lustre: atlas1-MDT0000: haven't heard from client f6198c53-52c9-efdc-5a22-78da69efaaee (at 9306@gni100) in 920 seconds. I think it's dead, and I am evicting it. exp ffff883f81f8b000, cur 1459804615 expire 1459803715 last 1459803695 <4>[3531983.201025] Lustre: atlas1-MDT0000: haven't heard from client 3ef2c4ae-9837-5b08-1bd4-867156f20457 (at 9311@gni100) in 1350 seconds. I think it's dead, and I am evicting it. exp ffff883f3b778800, cur 1459805066 expire 1459804166 last 1459803716 <3>[3574278.423701] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9316@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883ad51cb780/0xe00c22c8c284d5be lrc: 4/0,0 mode: PR/PR res: [0x2001854b9:0x22b8:0x0].0 bits 0x13 rrc: 11 type: IBT flags: 0x60200000000020 nid: 9316@gni100 remote: 0xbdb01e76e0a55526 expref: 121 pid: 16070 timeout: 7868947574 lvb_type: 0 <3>[3574278.466975] LustreError: 15815:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9316@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883ad51cb780/0xe00c22c8c284d5be lrc: 4/0,0 mode: PR/PR res: [0x2001854b9:0x22b8:0x0].0 bits 0x13 rrc: 11 type: IBT flags: 0x60200000000020 nid: 9316@gni100 remote: 0xbdb01e76e0a55526 expref: 118 pid: 16070 timeout: 7868947574 lvb_type: 0 <6>[3582720.613380] Lustre: atlas1-MDT0000-mdc-ffff883f3134e800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3582720.613518] Lustre: client wants to enable acl, but mdt not! <6>[3582720.613520] Lustre: Layout lock feature supported. <6>[3582720.637138] Lustre: Skipped 1008 previous similar messages <4>[3582720.753953] Lustre: Mounted atlas1-client <4>[3585241.396380] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3585244.093022] Lustre: Unmounted atlas1-client <4>[3585245.417410] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3585247.440599] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[3587485.961950] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 24 seconds <3>[3587485.973892] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.230.231@o2ib200 (124): c: 0, oc: 0, rc: 63 <3>[3587486.962862] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 12 seconds <3>[3587486.974824] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Skipped 1 previous similar message <3>[3587486.986739] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.230.237@o2ib200 (112): c: 0, oc: 0, rc: 63 <3>[3587487.001528] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Skipped 1 previous similar message <3>[3587511.962192] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 12 seconds <3>[3587511.974136] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.230.233@o2ib200 (112): c: 0, oc: 0, rc: 63 <3>[3587536.961761] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 18 seconds <3>[3587536.973692] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Skipped 1 previous similar message <3>[3587536.985664] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.230.234@o2ib200 (118): c: 0, oc: 0, rc: 63 <3>[3587537.000417] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Skipped 1 previous similar message <3>[3587562.961754] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 20 seconds <3>[3587562.973670] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.230.235@o2ib200 (120): c: 0, oc: 0, rc: 63 <3>[3587946.412415] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 9298@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff882e5cbd9c80/0xe00c22c8ff28c362 lrc: 4/0,0 mode: PR/PR res: [0x2002cbde0:0x12f7b:0x0].0 bits 0x13 rrc: 10 type: IBT flags: 0x60200000000020 nid: 9298@gni100 remote: 0x2a49f0f068ec4e5a expref: 478804 pid: 15903 timeout: 7882615940 lvb_type: 0 <3>[3587946.456067] LustreError: 15697:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 82@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff88397ede5380/0xe00c22c83f7570ed lrc: 4/0,0 mode: PR/PR res: [0x2002cbde0:0x12f7b:0x0].0 bits 0x13 rrc: 10 type: IBT flags: 0x60200000000020 nid: 82@gni100 remote: 0x4179285f4bc865e expref: 634408 pid: 15958 timeout: 7882615940 lvb_type: 0 <3>[3587947.397247] LustreError: 15697:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 83@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff88233b325740/0xe00c22c8a4d9ebf2 lrc: 4/0,0 mode: PR/PR res: [0x2002cbde0:0x12f7b:0x0].0 bits 0x13 rrc: 8 type: IBT flags: 0x60200000000020 nid: 83@gni100 remote: 0xca7c230548f2f776 expref: 329324 pid: 15355 timeout: 7882615940 lvb_type: 0 <4>[3588273.977221] LNet: No route to 12345-12@gni100 via 10.36.226.117@o2ib200 (all routers down) <4>[3588273.987160] Lustre: 16062:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1459861356/real 1459861356] req@ffff883739d58c00 x1527039995027600/t0(0) o104->atlas1-MDT0000@12@gni100:15/16 lens 296/224 e 0 to 1 dl 1459861923 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 <3>[3588274.018787] LustreError: 16062:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 12@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883a0924cb00/0xe00c22c8fbf7d36b lrc: 4/0,0 mode: PR/PR res: [0x2002cbf38:0x1c4bd:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x60200000000020 nid: 12@gni100 remote: 0xedfcf6da318f0a69 expref: 566719 pid: 15354 timeout: 7883318566 lvb_type: 0 <3>[3588274.060588] LustreError: 16062:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 1 previous similar message <4>[3588365.897333] LNet: No route to 12345-94@gni100 via 10.36.226.117@o2ib200 (all routers down) <4>[3588365.907234] Lustre: 15500:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1459861448/real 1459861448] req@ffff8836d4cd7c00 x1527039995129624/t0(0) o104->atlas1-MDT0000@94@gni100:15/16 lens 296/224 e 0 to 1 dl 1459862015 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 <3>[3588365.938870] LustreError: 15500:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 94@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8820a2ac9d40/0xe00c22b870e3540b lrc: 4/0,0 mode: PR/PR res: [0x20027c4a8:0x9bee:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 94@gni100 remote: 0xa3796e09ff7325d9 expref: 613763 pid: 15600 timeout: 7883410486 lvb_type: 0 <4>[3588368.287689] LNet: No route to 12345-94@gni100 via 10.36.226.117@o2ib200 (all routers down) <4>[3588368.297570] LNet: Skipped 17 previous similar messages <4>[3588368.303758] Lustre: 15557:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1459861451/real 1459861451] req@ffff880c948ec400 x1527039995132900/t0(0) o104->atlas1-MDT0000@94@gni100:15/16 lens 296/224 e 0 to 1 dl 1459862018 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 <4>[3588368.335369] Lustre: 15557:0:(client.c:1943:ptlrpc_expire_one_request()) Skipped 17 previous similar messages <4>[3588469.651250] LNet: No route to 12345-9310@gni100 via 10.36.226.117@o2ib200 (all routers down) <4>[3588469.661350] LNet: Skipped 8 previous similar messages <4>[3588469.667491] Lustre: 16015:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1459861552/real 1459861552] req@ffff88143c2e7000 x1527039995227792/t0(0) o104->atlas1-MDT0000@9310@gni100:15/16 lens 296/224 e 0 to 1 dl 1459862119 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 <4>[3588469.699346] Lustre: 16015:0:(client.c:1943:ptlrpc_expire_one_request()) Skipped 8 previous similar messages <3>[3588469.710923] LustreError: 16015:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9310@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8810f3da5880/0xe00c22c814e8ecbf lrc: 4/0,0 mode: PR/PR res: [0x2002ca205:0x11970:0x0].0 bits 0x13 rrc: 8 type: IBT flags: 0x60200000000020 nid: 9310@gni100 remote: 0xde91ac9c49a466a0 expref: 195361 pid: 16043 timeout: 7883514240 lvb_type: 0 <3>[3588469.753029] LustreError: 16015:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 26 previous similar messages <4>[3588504.568986] LNet: No route to 12345-9228@gni100 via 10.36.226.117@o2ib200 (all routers down) <4>[3588504.579063] LNet: Skipped 1 previous similar message <4>[3588504.585043] Lustre: 16177:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1459861587/real 1459861587] req@ffff88365b17a400 x1527039995286708/t0(0) o104->atlas1-MDT0000@9228@gni100:15/16 lens 296/224 e 0 to 1 dl 1459862154 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 <4>[3588504.616853] Lustre: 16177:0:(client.c:1943:ptlrpc_expire_one_request()) Skipped 1 previous similar message <3>[3588504.628329] LustreError: 16177:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9228@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8823efa273c0/0xe00c22c8e2323e8e lrc: 4/0,0 mode: PR/PR res: [0x2002cbb10:0xa2cb:0x0].0 bits 0x13 rrc: 13 type: IBT flags: 0x60200000000020 nid: 9228@gni100 remote: 0x5b5dccd39dbebd3 expref: 513438 pid: 15433 timeout: 7883549158 lvb_type: 0 <3>[3588504.670271] LustreError: 16177:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 1 previous similar message <4>[3588655.127747] LNet: No route to 12345-90@gni100 via 10.36.226.117@o2ib200 (all routers down) <4>[3588655.137636] LNet: Skipped 1 previous similar message <4>[3588655.143673] Lustre: 15725:0:(client.c:1943:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1459861737/real 1459861737] req@ffff8837e4092800 x1527039995433716/t0(0) o104->atlas1-MDT0000@90@gni100:15/16 lens 296/224 e 0 to 1 dl 1459862304 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 <4>[3588655.175278] Lustre: 15725:0:(client.c:1943:ptlrpc_expire_one_request()) Skipped 1 previous similar message <3>[3588655.186759] LustreError: 15725:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 90@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883b0a2a2280/0xe00c22c8ac8f2598 lrc: 4/0,0 mode: PR/PR res: [0x2000fb922:0x8f13:0x0].0 bits 0x13 rrc: 12 type: IBT flags: 0x60200000000020 nid: 90@gni100 remote: 0x71fd8eeae6a3c4d6 expref: 441962 pid: 15860 timeout: 7883699716 lvb_type: 0 <3>[3588655.228472] LustreError: 15725:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 1 previous similar message <4>[3588679.155942] Lustre: atlas1-MDT0000: haven't heard from client 8740208a-4dab-b35e-0f24-cefeeb9e628e (at 9228@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7ee4fc00, cur 1459861762 expire 1459860862 last 1459860410 <4>[3588679.181296] Lustre: Skipped 1 previous similar message <4>[3588679.973577] Lustre: atlas1-MDT0000: haven't heard from client 5a9c0844-cc11-f393-f6e5-a70bd2e078fd (at 9306@gni100) in 1346 seconds. I think it's dead, and I am evicting it. exp ffff883f42648800, cur 1459861762 expire 1459860862 last 1459860416 <4>[3588680.994127] Lustre: atlas1-MDT0000: haven't heard from client 140f862c-4648-2cef-0a40-d3b7019c683f (at 16050@gni100) in 1341 seconds. I think it's dead, and I am evicting it. exp ffff883f7e7e6800, cur 1459861763 expire 1459860862 last 1459860422 <4>[3588681.019542] Lustre: Skipped 13 previous similar messages <4>[3588684.158843] Lustre: atlas1-MDT0000: haven't heard from client d3e22e54-9e4d-4490-ae39-7b67f26df198 (at 15747@gni100) in 1320 seconds. I think it's dead, and I am evicting it. exp ffff883f7d159800, cur 1459861767 expire 1459860862 last 1459860447 <4>[3588684.184439] Lustre: Skipped 331 previous similar messages <3>[3590592.960325] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 15 seconds <3>[3590592.972252] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.230.236@o2ib200 (51): c: 0, oc: 0, rc: 63 <4>[3597710.142068] Lustre: atlas1-MDT0000: haven't heard from client 1efb20d3-3e51-da31-7ee6-ff0a31028f11 (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f42648000, cur 1459870793 expire 1459869893 last 1459869441 <4>[3597710.168023] Lustre: Skipped 18217 previous similar messages <4>[3598161.147938] Lustre: atlas1-MDT0000: haven't heard from client 1364609e-1e00-4f75-cf1d-e46563329cf8 (at 10.36.247.160@o2ib) in 1094 seconds. I think it's dead, and I am evicting it. exp ffff883fa4d17400, cur 1459871244 expire 1459870344 last 1459870150 <4>[3599639.448189] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3599643.200526] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3599871.145407] Lustre: atlas1-MDT0000: haven't heard from client 749844ed-7014-6e0e-2074-9c96d2019f77 (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f30c71800, cur 1459872954 expire 1459872054 last 1459871602 <4>[3604797.137171] Lustre: atlas1-MDT0000: haven't heard from client 1d2f504d-e3dd-59e5-5466-b1d4953c5db5 (at 10.38.144.32@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f33330c00, cur 1459877880 expire 1459876980 last 1459876528 <4>[3614038.952196] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3614042.637927] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3614044.652170] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3615562.138470] Lustre: atlas1-MDT0000: haven't heard from client 488c082d-22ca-cb83-216d-3bb35fe0440d (at 10.38.144.20@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4770b400, cur 1459888645 expire 1459887745 last 1459887293 <4>[3615562.164426] Lustre: Skipped 6 previous similar messages <4>[3624333.120338] Lustre: atlas1-MDT0000: haven't heard from client 51dfd192-5502-d83a-0721-a63bc7e126bb (at 8978@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3daa4800, cur 1459897416 expire 1459896516 last 1459896064 <4>[3625289.119726] Lustre: atlas1-MDT0000: haven't heard from client 50a1470f-8c32-ee40-92ae-89b6292d48b4 (at 8905@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883decf0a000, cur 1459898372 expire 1459897472 last 1459897020 <4>[3625289.145101] Lustre: Skipped 11 previous similar messages <4>[3625740.120399] Lustre: atlas1-MDT0000: haven't heard from client 4d0f6b04-d36e-ee6d-475f-97ff94f7d749 (at 8978@gni100) in 1299 seconds. I think it's dead, and I am evicting it. exp ffff883f34d03400, cur 1459898823 expire 1459897923 last 1459897524 <4>[3625740.145708] Lustre: Skipped 13 previous similar messages <4>[3625928.586710] Lustre: atlas1-MDT0000: Client 6555c15d-a4be-a38d-48ed-5f2c96419481 (at 12963@gni100) reconnecting <4>[3625961.884081] Lustre: atlas1-MDT0000: Client 1365b236-d789-65dd-01b3-7b46368a95d5 (at 8160@gni100) reconnecting <4>[3626061.881780] Lustre: atlas1-MDT0000: Client 99aa2a50-a71e-8157-9156-d40b6b35fc6a (at 7067@gni100) reconnecting <4>[3626068.393422] Lustre: atlas1-MDT0000: Client e2c72464-1bfb-b92f-456c-00fd599e8fe5 (at 8780@gni100) reconnecting <4>[3626097.039398] Lustre: atlas1-MDT0000: Client 1304a279-7fce-7457-81e5-4676c79e5981 (at 13657@gni100) reconnecting <4>[3626191.119268] Lustre: atlas1-MDT0000: haven't heard from client 9b891e37-21e3-f636-47ce-db7b553e63bf (at 10790@gni100) in 1238 seconds. I think it's dead, and I am evicting it. exp ffff883f3ad8a400, cur 1459899274 expire 1459898374 last 1459898036 <4>[3626191.150094] Lustre: Skipped 3 previous similar messages <4>[3626280.807293] Lustre: atlas1-MDT0000: Client af259c49-4c6c-a782-0788-32cda58cead3 (at 9174@gni100) reconnecting <4>[3626406.468791] Lustre: atlas1-MDT0000: Client ead39433-5165-551f-fb3a-bc73d22795d5 (at 8297@gni100) reconnecting <4>[3626444.165983] Lustre: atlas1-MDT0000: Client 389378b8-80b7-8cd6-69db-c195cfafb0e9 (at 14014@gni100) reconnecting <4>[3626619.898483] Lustre: atlas1-MDT0000: Client 78fed15b-1a54-86ad-0183-cc10270989df (at 10119@gni100) reconnecting <4>[3626619.910329] Lustre: Skipped 3 previous similar messages <4>[3626642.141203] Lustre: atlas1-MDT0000: haven't heard from client 10dfd6ec-2148-d317-195f-fb685d79db86 (at 3920@gni100) in 1155 seconds. I think it's dead, and I am evicting it. exp ffff883f46609000, cur 1459899725 expire 1459898825 last 1459898570 <4>[3626642.166490] Lustre: Skipped 2 previous similar messages <4>[3627007.159115] Lustre: atlas1-MDT0000: Client 509cbc13-c313-e6ba-bfc6-a261bed5a392 (at 9177@gni100) reconnecting <4>[3627329.434612] Lustre: atlas1-MDT0000: Client 6baa44b9-a176-c6a6-7725-c82582f1ac18 (at 103@gni100) reconnecting <4>[3627808.117619] Lustre: atlas1-MDT0000: haven't heard from client 94e2de8f-d87e-0104-16d8-ed190aa009e0 (at 12240@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3fb4fc00, cur 1459900891 expire 1459899991 last 1459899539 <4>[3627808.143034] Lustre: Skipped 1 previous similar message <4>[3628071.833909] Lustre: atlas1-MDT0000: Client cb6c5178-5e76-7c99-6adf-b4b191bff4ce (at 6085@gni100) reconnecting <4>[3628071.845669] Lustre: Skipped 8 previous similar messages <3>[3628496.378740] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9192@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff88242a8102c0/0xe00c22c99bb649d1 lrc: 4/0,0 mode: PR/PR res: [0x200259393:0x2574:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 9192@gni100 remote: 0x450a74899d4b60f2 expref: 10 pid: 15604 timeout: 7923165154 lvb_type: 0 <3>[3628496.421828] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) Skipped 2 previous similar messages <3>[3628496.433574] LustreError: 15562:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9192@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff88242a8102c0/0xe00c22c99bb649d1 lrc: 2/0,0 mode: --/PR res: [0x200259393:0x2574:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x44a01000000020 nid: 9192@gni100 remote: 0x450a74899d4b60f2 expref: 5 pid: 15604 timeout: 7923165154 lvb_type: 0 <3>[3628496.475158] LustreError: 15562:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) Skipped 14 previous similar messages <3>[3629073.378284] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 7571@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883ee204e580/0xe00c22c99e4d5383 lrc: 4/0,0 mode: PR/PR res: [0x200259393:0x2574:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x60200000000020 nid: 7571@gni100 remote: 0x76c93232d8af970d expref: 10 pid: 15549 timeout: 7923742395 lvb_type: 0 <3>[3629073.421411] LustreError: 15599:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 7571@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883ee204e580/0xe00c22c99e4d5383 lrc: 2/0,0 mode: --/PR res: [0x200259393:0x2574:0x0].0 bits 0x13 rrc: 6 type: IBT flags: 0x44a01000000020 nid: 7571@gni100 remote: 0x76c93232d8af970d expref: 4 pid: 15549 timeout: 7923742395 lvb_type: 0 <4>[3631059.782747] Lustre: atlas1-MDT0000: Client ea6f914e-8649-13a7-ed18-42b035f4bc99 (at 13876@gni100) reconnecting <4>[3631059.794605] Lustre: Skipped 3 previous similar messages <4>[3632879.695279] Lustre: atlas1-MDT0000: Client a41d223f-a536-7c89-8b7d-21c6d7452e89 (at 13713@gni100) reconnecting <4>[3635606.110920] Lustre: atlas1-MDT0000: haven't heard from client f7fd78d0-5820-0e78-3e8a-69d1cb313233 (at 9888@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f32c72400, cur 1459908689 expire 1459907789 last 1459907337 <4>[3638219.108707] Lustre: atlas1-MDT0000: haven't heard from client 8e9c55d2-0763-1057-8420-b001cc3c028e (at 14687@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f385f9400, cur 1459911302 expire 1459910402 last 1459909950 <4>[3639324.717151] Lustre: atlas1-MDT0000: Client 635cb1fb-834c-6daa-1326-1a03dbdbaed8 (at 14721@gni100) reconnecting <4>[3641797.105330] Lustre: atlas1-MDT0000: haven't heard from client 91209714-bd7f-45c2-ab75-412fa3da25ef (at 13412@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f451e2800, cur 1459914880 expire 1459913980 last 1459913528 <4>[3643881.103448] Lustre: atlas1-MDT0000: haven't heard from client f2ce16d0-1ff5-3076-5d50-13027f86a131 (at 8700@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f465fe800, cur 1459916964 expire 1459916064 last 1459915612 <4>[3656788.638618] Lustre: atlas1-MDT0000: Client 5d991753-feeb-1e25-8ed3-dbca9065a17e (at 11437@gni100) reconnecting <4>[3656906.091393] Lustre: atlas1-MDT0000: haven't heard from client 45ac1801-888d-c36e-6e81-5cbcc5df0931 (at 4460@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4846c400, cur 1459929989 expire 1459929089 last 1459928637 <4>[3666953.081859] Lustre: atlas1-MDT0000: haven't heard from client a235936c-4f78-c68f-a4e8-b10254c4b354 (at 1644@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f33fc4800, cur 1459940036 expire 1459939136 last 1459938684 <4>[3670294.078972] Lustre: atlas1-MDT0000: haven't heard from client 49157533-036d-01d7-1776-df80a229209a (at 4259@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883cca655000, cur 1459943377 expire 1459942477 last 1459942025 <4>[3671641.292554] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3671645.484684] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3671647.707455] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3675362.270084] Lustre: atlas1-MDT0000: Client 3be03200-aa0f-be48-b66d-d1df1632a1fe (at 10263@gni100) reconnecting <4>[3683150.066948] Lustre: atlas1-MDT0000: haven't heard from client e0265453-17bc-0cb2-d3df-03ace498c2b5 (at 1936@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f30cbac00, cur 1459956233 expire 1459955333 last 1459954881 <4>[3684369.065609] Lustre: atlas1-MDT0000: haven't heard from client 54d253fc-f8c0-1c74-d878-0e8f34a80c26 (at 14498@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3089bc00, cur 1459957452 expire 1459956552 last 1459956100 <4>[3686037.295791] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3686040.973441] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <6>[3687120.747073] Lustre: atlas1-MDT0000-mdc-ffff881161037000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3687120.747241] Lustre: client wants to enable acl, but mdt not! <6>[3687120.747243] Lustre: Layout lock feature supported. <6>[3687120.770829] Lustre: Skipped 1008 previous similar messages <4>[3687120.889192] Lustre: Mounted atlas1-client <4>[3689511.444415] Lustre: Unmounted atlas1-client <4>[3700444.201391] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3700447.857763] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3700558.052311] Lustre: atlas1-MDT0000: haven't heard from client 2ca7dda3-2085-8afc-8c92-c9e882e95502 (at 11367@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f81a01c00, cur 1459973641 expire 1459972741 last 1459972289 <4>[3701494.050442] Lustre: atlas1-MDT0000: haven't heard from client 59eadb76-dc22-60d2-42cb-7cb587142428 (at 10.36.207.135@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f48dda000, cur 1459974577 expire 1459973677 last 1459973225 <4>[3701494.076410] Lustre: Skipped 1 previous similar message <4>[3729458.027129] Lustre: atlas1-MDT0000: haven't heard from client 031a891e-d7df-1b2f-2b5d-4c7e337f1cf9 (at 8806@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f424e8800, cur 1460002541 expire 1460001641 last 1460001189 <4>[3733522.021482] Lustre: atlas1-MDT0000: haven't heard from client 5fbeb9ff-b38d-1237-d3b6-7869d982be05 (at 11049@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8801cbc95400, cur 1460006605 expire 1460005705 last 1460005253 <4>[3733522.046859] Lustre: Skipped 1 previous similar message <4>[3734762.020843] Lustre: atlas1-MDT0000: haven't heard from client 7914e14c-76d4-05e8-4d89-3bd99f7bfa25 (at 13620@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3ebf1400, cur 1460007845 expire 1460006945 last 1460006493 <6>[3755520.381200] Lustre: atlas1-MDT0000-mdc-ffff883738dfdc00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3755520.381349] Lustre: client wants to enable acl, but mdt not! <6>[3755520.381351] Lustre: Layout lock feature supported. <6>[3755520.405055] Lustre: Skipped 1008 previous similar messages <4>[3755520.524852] Lustre: Mounted atlas1-client <4>[3757869.245757] Lustre: Unmounted atlas1-client <4>[3758045.327958] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3758049.279676] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3761357.994702] Lustre: atlas1-MDT0000: haven't heard from client e8be2c75-6fa7-6c20-aeda-33208d1f7d7f (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f46730400, cur 1460034441 expire 1460033541 last 1460033089 <4>[3768291.987870] Lustre: atlas1-MDT0000: haven't heard from client 0e055a08-b697-fa5b-3642-e3387cc0ae28 (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3badc400, cur 1460041375 expire 1460040475 last 1460040023 <4>[3769663.986778] Lustre: atlas1-MDT0000: haven't heard from client dbac3580-cf98-bae9-05d3-5ea120b27a95 (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880d3be46400, cur 1460042747 expire 1460041847 last 1460041395 <4>[3769664.012755] Lustre: Skipped 1 previous similar message <4>[3770114.986333] Lustre: atlas1-MDT0000: haven't heard from client 7b4aafd9-4e09-68f8-ad3f-6e61a1048347 (at 15421@gni100) in 1104 seconds. I think it's dead, and I am evicting it. exp ffff8835a7af4400, cur 1460043198 expire 1460042298 last 1460042094 <4>[3772437.751601] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3772441.436436] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3786837.850127] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3786841.579891] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3817024.945845] Lustre: atlas1-MDT0000: haven't heard from client 6555c15d-a4be-a38d-48ed-5f2c96419481 (at 12963@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f484cf000, cur 1460090108 expire 1460089208 last 1460088756 <4>[3822423.939141] Lustre: atlas1-MDT0000: haven't heard from client 7ebadb86-c6bc-b307-f72b-eec619fa9f53 (at 9779@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3134ec00, cur 1460095507 expire 1460094607 last 1460094155 <6>[3841919.980753] Lustre: atlas1-MDT0000-mdc-ffff883ade319c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3841919.980878] Lustre: client wants to enable acl, but mdt not! <6>[3841919.980879] Lustre: Layout lock feature supported. <6>[3841920.009822] Lustre: Skipped 1008 previous similar messages <4>[3841920.122166] Lustre: Mounted atlas1-client <4>[3844322.278923] Lustre: Unmounted atlas1-client <4>[3844440.895383] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3844444.497216] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[3850131.959158] LustreError: 138-a: atlas1-MDT0000: A client on nid 9228@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[3850577.913749] Lustre: atlas1-MDT0000: haven't heard from client ef2fd47e-04e1-8e31-95db-b3d8e2517118 (at 10.38.144.26@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3a0a7000, cur 1460123661 expire 1460122761 last 1460122309 <4>[3856547.908706] Lustre: atlas1-MDT0000: haven't heard from client 4b3eea43-df0c-c479-8c41-57a61135fc82 (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8819a6de8000, cur 1460129631 expire 1460128731 last 1460128279 <4>[3858473.907460] Lustre: atlas1-MDT0000: haven't heard from client 662f1908-5964-9702-075e-4df6595d008f (at 10.36.207.134@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88119313b400, cur 1460131557 expire 1460130657 last 1460130205 <4>[3858838.247333] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3858841.937316] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3860506.905077] Lustre: atlas1-MDT0000: haven't heard from client 1fb7e15e-ac82-8795-6bfb-ecc3f1e7bb4d (at 11666@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f43615800, cur 1460133590 expire 1460132690 last 1460132238 <4>[3873238.402459] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3873242.017680] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3895190.431530] Lustre: atlas1-MDT0000: Client 57a6d637-9f2d-f76d-9322-c2ce7384a039 (at 2499@gni100) reconnecting <4>[3905940.290727] Lustre: atlas1-MDT0000: Client da82168f-e7da-5ced-1573-d5d52637c6cf (at 4905@gni100) reconnecting <4>[3912172.809864] Lustre: atlas1-MDT0000: Client aabca04a-9658-a1ac-fc0c-2ad55b41455f (at 1921@gni100) reconnecting <4>[3912172.821599] Lustre: Skipped 1 previous similar message <6>[3928320.241351] Lustre: atlas1-MDT0000-mdc-ffff881b3e675400: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[3928320.241460] Lustre: client wants to enable acl, but mdt not! <6>[3928320.241462] Lustre: Layout lock feature supported. <6>[3928320.265200] Lustre: Skipped 1008 previous similar messages <4>[3928320.381616] Lustre: Mounted atlas1-client <4>[3930758.326967] Lustre: Unmounted atlas1-client <4>[3930851.381843] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3930855.093269] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3930857.132835] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3934041.839992] Lustre: atlas1-MDT0000: haven't heard from client 0ba6c7b0-5f4a-1fcb-a8d4-58330980926b (at 18230@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f408b3c00, cur 1460207125 expire 1460206225 last 1460205773 <4>[3934041.865411] Lustre: Skipped 1 previous similar message <4>[3941113.928999] Lustre: atlas1-MDT0000: Client c1cd31b1-0be7-ad59-991a-da4efbc8140a (at 13791@gni100) reconnecting <4>[3941181.843051] Lustre: atlas1-MDT0000: haven't heard from client 407872b4-5ee4-1a93-59cf-ab1fb8216574 (at 18751@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f455c0000, cur 1460214265 expire 1460213365 last 1460212913 <4>[3945240.582649] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3945244.212843] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3954154.824720] Lustre: atlas1-MDT0000: haven't heard from client 95063d9a-9b76-69b2-b21a-75982fd6c10c (at 8789@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f42f4b400, cur 1460227238 expire 1460226338 last 1460225886 <4>[3954605.822934] Lustre: atlas1-MDT0000: haven't heard from client e128e66a-f224-30c9-6024-fae922fb31b5 (at 12689@gni100) in 1295 seconds. I think it's dead, and I am evicting it. exp ffff883f4766d400, cur 1460227689 expire 1460226789 last 1460226394 <4>[3959639.509250] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[3959643.173212] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[3982194.058778] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 376s: evicting client at 9109@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883ea6f02a80/0xe00c22d050846eec lrc: 4/0,0 mode: PR/PR res: [0x2000bd4c9:0xa6f2:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 9109@gni100 remote: 0x9287091b651d4ec1 expref: 24 pid: 16016 timeout: 8276863586 lvb_type: 0 <3>[3982194.101877] LustreError: 15768:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9109@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883ea6f02a80/0xe00c22d050846eec lrc: 2/0,0 mode: --/PR res: [0x2000bd4c9:0xa6f2:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x44a01000000020 nid: 9109@gni100 remote: 0x9287091b651d4ec1 expref: 10 pid: 16016 timeout: 8276863586 lvb_type: 0 <4>[4004347.776169] Lustre: atlas1-MDT0000: haven't heard from client 8187b60d-a78f-43bd-65f4-c80a6a1bcae0 (at 4589@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f39fd0c00, cur 1460277431 expire 1460276531 last 1460276079 <6>[4014719.838586] Lustre: atlas1-MDT0000-mdc-ffff883f7da77000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4014719.838754] Lustre: client wants to enable acl, but mdt not! <6>[4014719.838756] Lustre: Layout lock feature supported. <6>[4014719.862336] Lustre: Skipped 1008 previous similar messages <4>[4014719.978462] Lustre: Mounted atlas1-client <4>[4017239.219036] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4017242.940104] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4017245.224667] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4017259.146196] Lustre: Unmounted atlas1-client <4>[4027849.758728] Lustre: atlas1-MDT0000: haven't heard from client e8cac7a9-c03f-d104-11f3-6091a9917149 (at 10524@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880ddd1a4000, cur 1460300933 expire 1460300033 last 1460299581 <4>[4031060.757281] Lustre: atlas1-MDT0000: haven't heard from client 842c2bab-6cfe-2c37-7152-ba45b9c826a9 (at 8358@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f491c8400, cur 1460304144 expire 1460303244 last 1460302792 <4>[4031060.782591] Lustre: Skipped 1 previous similar message <4>[4031637.510693] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4031641.149014] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4036748.778236] Lustre: atlas1-MDT0000: Client f54f523f-ed12-7e6b-2710-31592ae2db70 (at 5915@gni100) reconnecting <4>[4038910.749324] Lustre: atlas1-MDT0000: haven't heard from client dac8b85b-f9a6-d184-558d-f3b59ebe50bd (at 8471@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f327c6800, cur 1460311994 expire 1460311094 last 1460310642 <4>[4039361.745376] Lustre: atlas1-MDT0000: haven't heard from client ab137b30-10ac-ccb3-7cd1-18423a27a487 (at 5914@gni100) in 1089 seconds. I think it's dead, and I am evicting it. exp ffff883f477dc400, cur 1460312445 expire 1460311545 last 1460311356 <4>[4040473.745015] Lustre: atlas1-MDT0000: haven't heard from client f54f523f-ed12-7e6b-2710-31592ae2db70 (at 5915@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f426de000, cur 1460313557 expire 1460312657 last 1460312205 <4>[4046043.905583] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4046047.501632] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4071247.716572] Lustre: atlas1-MDT0000: haven't heard from client f7d3b0a6-1fec-1c8d-5e16-512eb407567f (at 9273@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883bd8bf1400, cur 1460344331 expire 1460343431 last 1460342979 <4>[4084254.844099] Lustre: atlas1-MDT0000: Client 88b0a016-87b1-575b-d420-54aa91961ea1 (at 3391@gni100) reconnecting <4>[4096666.694470] Lustre: atlas1-MDT0000: haven't heard from client 856a8c8b-4e39-ee11-ded1-e5e97af5b75e (at 14244@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f44c79c00, cur 1460369750 expire 1460368850 last 1460368398 <6>[4101120.264356] Lustre: atlas1-MDT0000-mdc-ffff883f45563c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4101120.264524] Lustre: client wants to enable acl, but mdt not! <6>[4101120.264526] Lustre: Layout lock feature supported. <6>[4101120.288103] Lustre: Skipped 1008 previous similar messages <4>[4101120.402651] Lustre: Mounted atlas1-client <4>[4103653.186977] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4103656.908540] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4103711.270501] Lustre: Unmounted atlas1-client <4>[4108942.681992] Lustre: atlas1-MDT0000: haven't heard from client 7d337af3-af4c-a74f-4128-a760782c5b67 (at 10.38.144.22@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883febba9800, cur 1460382026 expire 1460381126 last 1460380674 <4>[4108942.708002] Lustre: Skipped 1 previous similar message <3>[4116011.937337] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 9198@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff881167faeb40/0xe00c22d16bc1d4d6 lrc: 4/0,0 mode: PR/PR res: [0x2000bd4c9:0xa6f2:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x60200000000020 nid: 9198@gni100 remote: 0x8644ec969cfe9a55 expref: 43 pid: 15905 timeout: 8410681762 lvb_type: 0 <3>[4116011.980441] LustreError: 15294:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 9198@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff881167faeb40/0xe00c22d16bc1d4d6 lrc: 4/0,0 mode: PR/PR res: [0x2000bd4c9:0xa6f2:0x0].0 bits 0x13 rrc: 9 type: IBT flags: 0x60200000000020 nid: 9198@gni100 remote: 0x8644ec969cfe9a55 expref: 27 pid: 15905 timeout: 8410681762 lvb_type: 0 <4>[4118045.156512] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4118048.778500] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4129308.662529] Lustre: atlas1-MDT0000: haven't heard from client 8e84ed9e-dc65-503c-b3e9-87c4c8df14ea (at 10.36.207.152@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f43228400, cur 1460402392 expire 1460401492 last 1460401040 <4>[4129308.688495] Lustre: Skipped 1 previous similar message <4>[4132437.407316] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4132441.065486] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4134749.658644] Lustre: atlas1-MDT0000: haven't heard from client 5b607a85-66f8-9c54-2474-45cc0bfec01c (at 18040@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883937716000, cur 1460407833 expire 1460406933 last 1460406481 <4>[4154587.639494] Lustre: atlas1-MDT0000: haven't heard from client cb765c75-df5f-5c00-0145-83f24bad2030 (at 10.36.207.196@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f44c30400, cur 1460427671 expire 1460426771 last 1460426319 <4>[4159020.634857] Lustre: atlas1-MDT0000: haven't heard from client fb36b7fb-79a8-176c-03d3-cc40942fe356 (at 18062@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4847e400, cur 1460432104 expire 1460431204 last 1460430752 <4>[4160230.637574] Lustre: atlas1-MDT0000: haven't heard from client 732c6b44-df8a-d257-0a2a-f09706c7fe4a (at 13026@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff88373d21c000, cur 1460433314 expire 1460432414 last 1460431962 <4>[4161726.632589] Lustre: atlas1-MDT0000: haven't heard from client e1b73b74-0807-235f-883d-5c7578531d0d (at 12121@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8802bfe57800, cur 1460434810 expire 1460433910 last 1460433458 <4>[4166994.547509] Lustre: atlas1-MDT0000: Client 6c1921f1-58a6-042d-a29d-119bc67142f8 (at 18737@gni100) reconnecting <4>[4167648.627617] Lustre: atlas1-MDT0000: haven't heard from client 3352cfbc-b236-bb08-15db-81cf9096a5f3 (at 12179@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff880b4ca21c00, cur 1460440732 expire 1460439832 last 1460439380 <4>[4168099.627653] Lustre: atlas1-MDT0000: haven't heard from client 02b0ce5b-82f1-119a-ac3a-e649f0c5750b (at 5716@gni100) in 1186 seconds. I think it's dead, and I am evicting it. exp ffff883f3feb2000, cur 1460441183 expire 1460440283 last 1460439997 <4>[4176118.621506] Lustre: atlas1-MDT0000: haven't heard from client af373174-8b7a-d0be-2ccc-6b11e211af6d (at 12768@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f452e6000, cur 1460449202 expire 1460448302 last 1460447850 <4>[4177146.617000] Lustre: atlas1-MDT0000: haven't heard from client df49bca8-01da-7268-f4fd-4778e83784c9 (at 12353@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883eb5c86000, cur 1460450230 expire 1460449330 last 1460448878 <4>[4181230.947299] Lustre: atlas1-MDT0000: Client 43e56f65-7c45-a15e-fa6e-fc76a07ee97b (at 10705@gni100) reconnecting <6>[4187519.839384] Lustre: atlas1-MDT0000-mdc-ffff883f78064800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4187519.839562] Lustre: client wants to enable acl, but mdt not! <6>[4187519.839564] Lustre: Layout lock feature supported. <6>[4187519.863159] Lustre: Skipped 1008 previous similar messages <4>[4187519.980347] Lustre: Mounted atlas1-client <4>[4187871.607416] Lustre: atlas1-MDT0000: haven't heard from client 5a175a12-659c-9e61-a3b4-ec8eefc5799a (at 10.36.202.180@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883fb0cc1400, cur 1460460955 expire 1460460055 last 1460459603 <4>[4190040.531710] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4190044.506056] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4190112.263211] Lustre: Unmounted atlas1-client <4>[4204437.091958] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4204440.718079] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4208759.586425] Lustre: atlas1-MDT0000: haven't heard from client 86860cd1-9554-ca39-34bb-717a07e9c567 (at 1669@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4312a800, cur 1460481843 expire 1460480943 last 1460480491 <4>[4208759.611711] Lustre: Skipped 7 previous similar messages <4>[4209210.587919] Lustre: atlas1-MDT0000: haven't heard from client 7768ce06-e752-e238-a5ce-84f9451b8260 (at 10.36.207.196@o2ib) in 1145 seconds. I think it's dead, and I am evicting it. exp ffff883becc0a800, cur 1460482294 expire 1460481394 last 1460481149 <4>[4209661.584394] Lustre: atlas1-MDT0000: haven't heard from client 6b6ca7db-214b-dbbc-8d5c-56c41ba6c83d (at 10.36.207.196@o2ib) in 958 seconds. I think it's dead, and I am evicting it. exp ffff88236f03d800, cur 1460482745 expire 1460481845 last 1460481787 <4>[4218837.644834] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4218841.336093] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4218843.361272] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4219647.283141] Lustre: atlas1-MDT0000: Client e51ba279-3c66-a96a-47f2-ff8de4d07f5b (at 5894@gni100) reconnecting <4>[4222735.250602] Lustre: atlas1-MDT0000: Client 7d49d90c-d292-bb72-468c-82199d8a2cf7 (at 12235@gni100) reconnecting <4>[4222936.621784] Lustre: atlas1-MDT0000: Client a39d6377-5c0b-988b-3ebc-ba4f99bdc7e8 (at 12968@gni100) reconnecting <4>[4225849.569041] Lustre: atlas1-MDT0000: haven't heard from client 8bed8898-1ca5-6616-1066-8e31423abf41 (at 95@gni2) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3a0b3800, cur 1460498933 expire 1460498033 last 1460497581 <3>[4227068.379775] LNetError: 14804:0:(o2iblnd_cb.c:3018:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 22 seconds <3>[4227068.391781] LNetError: 14804:0:(o2iblnd_cb.c:3081:kiblnd_check_conns()) Timed out RDMA with 10.36.223.7@o2ib (50): c: 0, oc: 0, rc: 63 <4>[4235833.560350] Lustre: atlas1-MDT0000: haven't heard from client f8a3612b-e2db-5629-6e59-fc3539bac4db (at 9310@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f47c56c00, cur 1460508917 expire 1460508017 last 1460507565 <4>[4235833.585766] Lustre: Skipped 81 previous similar messages <3>[4237963.849860] LustreError: 138-a: atlas1-MDT0000: A client on nid 94@gni100 was evicted due to a lock blocking callback time out: rc -107 <4>[4238550.240115] Lustre: atlas1-MDT0000: Client 30da6f52-2c61-1112-9dd9-92c5c10d28aa (at 11722@gni100) reconnecting <4>[4256325.540471] Lustre: atlas1-MDT0000: haven't heard from client 9ce9148f-6e1b-ef28-57c0-232694fff083 (at 7721@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3b72b400, cur 1460529409 expire 1460528509 last 1460528057 <6>[4273919.327260] Lustre: atlas1-MDT0000-mdc-ffff883f466d3c00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4273919.327405] Lustre: client wants to enable acl, but mdt not! <6>[4273919.327406] Lustre: Layout lock feature supported. <6>[4273919.351009] Lustre: Skipped 1008 previous similar messages <4>[4273919.467994] Lustre: Mounted atlas1-client <4>[4276440.495747] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4276444.186138] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4276505.561018] Lustre: Unmounted atlas1-client <4>[4285133.513996] Lustre: atlas1-MDT0000: haven't heard from client 0bcdbbfb-3b6a-5ac8-b37f-449ffb9ef045 (at 10.36.207.196@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883d73ffa800, cur 1460558217 expire 1460557317 last 1460556865 <4>[4290836.754130] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4290840.322817] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4303168.279838] Lustre: atlas1-MDT0000: Client 15afddd3-0c6e-7b1c-b734-9ec3ad4e5575 (at 14864@gni100) reconnecting <4>[4304002.497655] Lustre: atlas1-MDT0000: haven't heard from client dabadb19-c608-3ae8-a7ff-3ca9bf4a89fa (at 16274@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f48661400, cur 1460577086 expire 1460576186 last 1460575734 <4>[4305242.787383] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4305246.412265] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4325878.476692] Lustre: atlas1-MDT0000: haven't heard from client 43b8c62d-5c61-44c7-7f2d-76580d20c00f (at 15132@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f30e41000, cur 1460598962 expire 1460598062 last 1460597610 <6>[4360319.655414] Lustre: atlas1-MDT0000-mdc-ffff883f43648000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4360319.655564] Lustre: client wants to enable acl, but mdt not! <6>[4360319.655566] Lustre: Layout lock feature supported. <6>[4360319.679176] Lustre: Skipped 1008 previous similar messages <4>[4360319.795886] Lustre: Mounted atlas1-client <4>[4362280.448161] Lustre: atlas1-MDT0000: haven't heard from client 7e74d13b-57be-657a-313a-a1f65c67a218 (at 14033@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3c1ca400, cur 1460635364 expire 1460634464 last 1460634012 <4>[4362840.597930] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4362844.335621] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4362846.386684] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4362888.933937] Lustre: Unmounted atlas1-client <4>[4366088.745840] Lustre: atlas1-MDT0000: Client 7e74d13b-57be-657a-313a-a1f65c67a218 (at 14033@gni100) reconnecting <4>[4370652.431561] Lustre: atlas1-MDT0000: haven't heard from client 80214717-b959-8aaa-9d5a-89b07645042b (at 4584@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3ac14000, cur 1460643736 expire 1460642836 last 1460642384 <4>[4376776.276864] Lustre: atlas1-MDT0000: Client 84d830a0-1fe7-a571-1a76-bad3f34f5fee (at 4712@gni100) reconnecting <4>[4377236.843980] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4377240.561376] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4377242.648018] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4386304.417612] Lustre: atlas1-MDT0000: haven't heard from client 1faa3a84-14dd-04c3-3805-90a43d1b3400 (at 15257@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f30912800, cur 1460659388 expire 1460658488 last 1460658036 <4>[4386304.443141] Lustre: Skipped 2 previous similar messages <4>[4391636.112391] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4391639.694070] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4391645.413080] Lustre: atlas1-MDT0000: haven't heard from client 8731da41-a096-7c88-b3e4-1156f9e398ec (at 10.36.207.196@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4232e400, cur 1460664729 expire 1460663829 last 1460663377 <4>[4394164.409435] Lustre: atlas1-MDT0000: haven't heard from client 375b38b5-7914-9587-7566-f89b02dd4137 (at 10.36.202.183@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883bbea4f400, cur 1460667248 expire 1460666348 last 1460665896 <4>[4394615.409116] Lustre: atlas1-MDT0000: haven't heard from client 2e95ae37-de2a-6804-6cd1-605fd93b1648 (at 10.36.202.180@o2ib) in 1142 seconds. I think it's dead, and I am evicting it. exp ffff883f47dc6400, cur 1460667699 expire 1460666799 last 1460666557 <4>[4394615.435080] Lustre: Skipped 7 previous similar messages <4>[4409228.396428] Lustre: atlas1-MDT0000: haven't heard from client b449591b-0328-5c03-d295-8930f726887f (at 14389@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f467e3000, cur 1460682312 expire 1460681412 last 1460680960 <4>[4409228.421850] Lustre: Skipped 7 previous similar messages <4>[4418276.386486] Lustre: atlas1-MDT0000: haven't heard from client f5171441-d3f0-bc18-ebac-43fae8406d89 (at 18142@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f44237400, cur 1460691360 expire 1460690460 last 1460690008 <4>[4428353.375312] Lustre: atlas1-MDT0000: haven't heard from client cbda2132-e3c0-2d7a-78dd-69eb8f30e54f (at 10.36.207.196@o2ib) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8826fdb4f800, cur 1460701437 expire 1460700537 last 1460700085 <4>[4429609.668692] Lustre: atlas1-MDT0000: Client bbd11b06-8668-446c-4909-416678502951 (at 10327@gni100) reconnecting <4>[4434683.210264] Lustre: atlas1-MDT0000: Client 26090e2f-551c-5577-b34f-ddb1c9f57240 (at 10.36.226.122@o2ib) reconnecting <6>[4446720.301173] Lustre: atlas1-MDT0000-mdc-ffff883f47c1bc00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4446720.301229] Lustre: client wants to enable acl, but mdt not! <6>[4446720.301231] Lustre: Layout lock feature supported. <6>[4446720.324928] Lustre: Skipped 1008 previous similar messages <4>[4446720.468786] Lustre: Mounted atlas1-client <3>[4447609.619219] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 6044@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff883e60909500/0xe00c22d740b168ba lrc: 4/0,0 mode: PR/PR res: [0x2002bd282:0x1a2:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x60200000000020 nid: 6044@gni100 remote: 0x14cc7c32f293fa21 expref: 7 pid: 15483 timeout: 8742279966 lvb_type: 0 <3>[4447609.662220] LustreError: 16097:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 6044@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff883e60909500/0xe00c22d740b168ba lrc: 2/0,0 mode: --/PR res: [0x2002bd282:0x1a2:0x0].0 bits 0x13 rrc: 5 type: IBT flags: 0x44a01000000020 nid: 6044@gni100 remote: 0x14cc7c32f293fa21 expref: 4 pid: 15483 timeout: 8742279966 lvb_type: 0 <4>[4449259.314087] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4449263.375430] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4449265.431641] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4449298.774089] Lustre: Unmounted atlas1-client <4>[4452058.374372] Lustre: atlas1-MDT0000: Client 90ea5f2d-50e3-dba2-47cc-f7491e7b015c (at 9825@gni100) reconnecting <4>[4452478.355042] Lustre: atlas1-MDT0000: haven't heard from client 889c1a06-abe0-5874-a887-df74c0182c67 (at 12715@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3d583000, cur 1460725562 expire 1460724662 last 1460724210 <4>[4456840.047138] Lustre: atlas1-MDT0000: Client c05157bd-4d6d-19d9-45a7-9769b56801d0 (at 5119@gni100) reconnecting <4>[4458047.401406] Lustre: atlas1-MDT0000: Client f0ca6076-1271-bbea-3029-9f4f4913edbc (at 1915@gni100) reconnecting <4>[4463317.342160] Lustre: atlas1-MDT0000: haven't heard from client 24fa6de7-787f-1c74-bf90-99a8aeee015d (at 5170@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3b093c00, cur 1460736401 expire 1460735501 last 1460735049 <4>[4463641.676353] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4463645.278358] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4478037.337854] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4478041.028443] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4481478.325034] Lustre: atlas1-MDT0000: haven't heard from client 7a064ea0-d389-413c-0d5b-b6a853ba4241 (at 14744@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f45529800, cur 1460754562 expire 1460753662 last 1460753210 <4>[4481478.350493] Lustre: Skipped 15 previous similar messages <4>[4519800.290292] Lustre: atlas1-MDT0000: haven't heard from client 317f20b8-5561-b9f7-cf9f-eb1b61980035 (at 8049@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f40ce3400, cur 1460792884 expire 1460791984 last 1460791532 <6>[4533119.228523] Lustre: atlas1-MDT0000-mdc-ffff8820125efc00: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4533119.228673] Lustre: client wants to enable acl, but mdt not! <6>[4533119.228675] Lustre: Layout lock feature supported. <6>[4533119.252271] Lustre: Skipped 1008 previous similar messages <4>[4533119.371225] Lustre: Mounted atlas1-client <4>[4535621.341735] Lustre: Unmounted atlas1-client <4>[4535648.097283] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4535651.738745] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4547700.264061] Lustre: atlas1-MDT0000: haven't heard from client b846c425-b656-8af2-e1a9-d67991de5b0b (at 17996@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3995a800, cur 1460820784 expire 1460819884 last 1460819432 <4>[4550041.743355] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4550045.434042] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4564445.307547] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4564448.912741] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4595770.489116] Lustre: atlas1-MDT0000: Client ebe307e8-af93-9298-0d69-96f8fa33cbf4 (at 9724@gni100) reconnecting <6>[4619519.804082] Lustre: atlas1-MDT0000-mdc-ffff883dede05000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4619519.804247] Lustre: client wants to enable acl, but mdt not! <6>[4619519.804249] Lustre: Layout lock feature supported. <6>[4619519.827865] Lustre: Skipped 1008 previous similar messages <4>[4619519.944446] Lustre: Mounted atlas1-client <4>[4622006.948684] Lustre: Unmounted atlas1-client <4>[4622046.547619] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4622050.214725] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4622052.377526] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <3>[4635011.448145] LustreError: 0:0:(ldlm_lockd.c:348:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 10727@gni100 ns: mdt-atlas1-MDT0000_UUID lock: ffff8836c3e85700/0xe00c22da64867af3 lrc: 4/0,0 mode: PR/PR res: [0x2001854b9:0x22b8:0x0].0 bits 0x13 rrc: 16 type: IBT flags: 0x60200000000020 nid: 10727@gni100 remote: 0xbe800309bfd8badc expref: 19 pid: 15824 timeout: 8929681929 lvb_type: 0 <3>[4635011.491599] LustreError: 15673:0:(ldlm_lockd.c:672:ldlm_handle_ast_error()) ### client (nid 10727@gni100) returned 0 from blocking AST ns: mdt-atlas1-MDT0000_UUID lock: ffff8836c3e85700/0xe00c22da64867af3 lrc: 2/0,0 mode: --/PR res: [0x2001854b9:0x22b8:0x0].0 bits 0x13 rrc: 16 type: IBT flags: 0x44a01000000020 nid: 10727@gni100 remote: 0xbe800309bfd8badc expref: 4 pid: 15824 timeout: 8929681929 lvb_type: 0 <4>[4636442.715953] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4636446.387959] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4636448.396626] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4650843.941240] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4650847.563700] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4658149.167910] Lustre: atlas1-MDT0000: haven't heard from client 65a2d0f4-b8f8-62c4-eba1-395ecb2dd4da (at 8236@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3ad75800, cur 1460931233 expire 1460930333 last 1460929881 <4>[4689277.137256] Lustre: atlas1-MDT0000: haven't heard from client 2d510ed4-e1cb-da62-c290-fda7570a5ea4 (at 4727@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f815a9400, cur 1460962361 expire 1460961461 last 1460961009 <4>[4697246.130722] Lustre: atlas1-MDT0000: haven't heard from client 26f0445c-aafa-228e-7c73-06e0548578b7 (at 10265@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3d53e800, cur 1460970330 expire 1460969430 last 1460968978 <6>[4705919.371202] Lustre: atlas1-MDT0000-mdc-ffff880228fba000: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4705919.371363] Lustre: client wants to enable acl, but mdt not! <6>[4705919.371364] Lustre: Layout lock feature supported. <6>[4705919.394975] Lustre: Skipped 1008 previous similar messages <4>[4705919.510383] Lustre: Mounted atlas1-client <4>[4708441.375682] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4708445.036448] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4708449.647179] Lustre: Unmounted atlas1-client <4>[4713309.118195] Lustre: atlas1-MDT0000: haven't heard from client 556f80a3-c64a-8583-86ac-d9f855fce05e (at 10.38.144.23@o2ib5) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f7eeed800, cur 1460986393 expire 1460985493 last 1460985041 <4>[4713760.114979] Lustre: atlas1-MDT0000: haven't heard from client fdeebc6b-c220-6cbb-b439-cca3dc30e601 (at 10.36.207.196@o2ib) in 974 seconds. I think it's dead, and I am evicting it. exp ffff883b45200c00, cur 1460986844 expire 1460985944 last 1460985870 <4>[4713760.140894] Lustre: Skipped 1 previous similar message <4>[4715595.167708] Lustre: atlas1-MDT0000: Client bad092de-6f9f-2f47-8f89-9cd7f17cf32d (at 18474@gni100) reconnecting <4>[4722619.106357] Lustre: atlas1-MDT0000: haven't heard from client 1ed55419-02d4-0d04-3ee5-9aec832f31f2 (at 5226@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f53b64400, cur 1460995703 expire 1460994803 last 1460994351 <4>[4722836.954794] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4722840.576418] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4732816.097357] Lustre: atlas1-MDT0000: haven't heard from client d0e563be-8b42-b739-35ef-d8c8f9692bf3 (at 12324@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f47d5b000, cur 1461005900 expire 1461005000 last 1461004548 <4>[4737236.169847] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4737239.909796] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4737242.067076] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4746249.084159] Lustre: atlas1-MDT0000: haven't heard from client d899ae45-fe7e-a59b-1b20-6c76c2df2e5d (at 7797@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f82616c00, cur 1461019333 expire 1461018433 last 1461017981 <4>[4749172.082531] Lustre: atlas1-MDT0000: haven't heard from client 6a728d92-2b98-1fd7-8f26-ae58ff688bc0 (at 2949@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f34134000, cur 1461022256 expire 1461021356 last 1461020904 <4>[4752496.079781] Lustre: atlas1-MDT0000: haven't heard from client 743112b6-9e51-be85-f392-6661064d4833 (at 8060@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883dc7c15800, cur 1461025580 expire 1461024680 last 1461024228 <4>[4752947.079355] Lustre: atlas1-MDT0000: haven't heard from client 20185bdb-8d0c-e56e-ef8f-417d45f06d0c (at 12531@gni100) in 937 seconds. I think it's dead, and I am evicting it. exp ffff880318d53400, cur 1461026031 expire 1461025131 last 1461025094 <4>[4771241.062760] Lustre: atlas1-MDT0000: haven't heard from client d48dd440-d6cd-562c-b52f-82a57b3701b8 (at 1096@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883ff1a23000, cur 1461044325 expire 1461043425 last 1461042973 <4>[4780621.055565] Lustre: atlas1-MDT0000: haven't heard from client f074cad1-12c9-d48b-3d62-1c124700a061 (at 304@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4615e000, cur 1461053705 expire 1461052805 last 1461052353 <6>[4792319.700751] Lustre: atlas1-MDT0000-mdc-ffff883a57924800: Connection restored to atlas1-MDT0000 (at 0@lo) <6>[4792319.700870] Lustre: client wants to enable acl, but mdt not! <6>[4792319.700873] Lustre: Layout lock feature supported. <6>[4792319.724517] Lustre: Skipped 1008 previous similar messages <4>[4792319.840156] Lustre: Mounted atlas1-client <4>[4794419.732705] ptlrpcd_0: page allocation failure. order:1, mode:0x20 <4>[4794419.733196] ptlrpcd_2: page allocation failure. order:1, mode:0x20 <4>[4794419.733198] Pid: 14890, comm: ptlrpcd_2 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[4794419.733199] Call Trace: <4>[4794419.733205] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[4794419.733214] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[4794419.733217] [] ? kmem_getpages+0x62/0x170 <4>[4794419.733218] [] ? fallback_alloc+0x1ba/0x270 <4>[4794419.733220] [] ? cache_grow+0x2cf/0x320 <4>[4794419.733221] [] ? ____cache_alloc_node+0x99/0x160 <4>[4794419.733223] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[4794419.733225] [] ? __kmalloc+0x199/0x230 <4>[4794419.733226] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[4794419.733228] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[4794419.733258] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[4794419.733276] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[4794419.733293] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[4794419.733309] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[4794419.733329] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[4794419.733346] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[4794419.733349] [] ? default_wake_function+0x0/0x20 <4>[4794419.733367] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[4794419.733370] [] ? kthread+0x9e/0xc0 <4>[4794419.733372] [] ? child_rip+0xa/0x20 <4>[4794419.733373] [] ? kthread+0x0/0xc0 <4>[4794419.733375] [] ? child_rip+0x0/0x20 <6>[4794419.733375] Mem-Info: <4>[4794419.733376] Node 0 DMA per-cpu: <4>[4794419.733377] CPU 0: hi: 0, btch: 1 usd: 0 <4>[4794419.733378] CPU 1: hi: 0, btch: 1 usd: 0 <4>[4794419.733378] CPU 2: hi: 0, btch: 1 usd: 0 <4>[4794419.733379] CPU 3: hi: 0, btch: 1 usd: 0 <4>[4794419.733380] CPU 4: hi: 0, btch: 1 usd: 0 <4>[4794419.733380] CPU 5: hi: 0, btch: 1 usd: 0 <4>[4794419.733381] CPU 6: hi: 0, btch: 1 usd: 0 <4>[4794419.733382] CPU 7: hi: 0, btch: 1 usd: 0 <4>[4794419.733382] Node 0 DMA32 per-cpu: <4>[4794419.733383] CPU 0: hi: 186, btch: 31 usd: 0 <4>[4794419.733384] CPU 1: hi: 186, btch: 31 usd: 0 <4>[4794419.733384] CPU 2: hi: 186, btch: 31 usd: 0 <4>[4794419.733385] CPU 3: hi: 186, btch: 31 usd: 0 <4>[4794419.733386] CPU 4: hi: 186, btch: 31 usd: 0 <4>[4794419.733386] CPU 5: hi: 186, btch: 31 usd: 0 <4>[4794419.733387] CPU 6: hi: 186, btch: 31 usd: 0 <4>[4794419.733388] CPU 7: hi: 186, btch: 31 usd: 0 <4>[4794419.733388] Node 0 Normal per-cpu: <4>[4794419.733389] CPU 0: hi: 186, btch: 31 usd: 133 <4>[4794419.733389] CPU 1: hi: 186, btch: 31 usd: 165 <4>[4794419.733390] CPU 2: hi: 186, btch: 31 usd: 162 <4>[4794419.733391] CPU 3: hi: 186, btch: 31 usd: 58 <4>[4794419.733391] CPU 4: hi: 186, btch: 31 usd: 169 <4>[4794419.733392] CPU 5: hi: 186, btch: 31 usd: 171 <4>[4794419.733392] CPU 6: hi: 186, btch: 31 usd: 156 <4>[4794419.733393] CPU 7: hi: 186, btch: 31 usd: 33 <4>[4794419.733395] active_anon:3410342 inactive_anon:157871 isolated_anon:0 <4>[4794419.733396] active_file:8915879 inactive_file:28807773 isolated_file:0 <4>[4794419.733396] unevictable:8901 dirty:4621 writeback:0 unstable:0 <4>[4794419.733396] free:189348 slab_reclaimable:9438421 slab_unreclaimable:14277731 <4>[4794419.733397] mapped:9211 shmem:214095 pagetables:7995 bounce:0 <4>[4794419.733397] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[4794419.733401] lowmem_reserve[]: 0 1880 258420 258420 <4>[4794419.733402] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[4794419.733406] lowmem_reserve[]: 0 0 256540 256540 <4>[4794419.733407] Node 0 Normal free:356536kB min:67088kB low:83860kB high:100632kB active_anon:13641368kB inactive_anon:631484kB active_file:35663516kB inactive_file:115231092kB unevictable:35604kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:18484kB writeback:0kB mapped:36844kB shmem:856380kB slab_reclaimable:37753684kB slab_unreclaimable:57110924kB kernel_stack:74624kB pagetables:31980kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[4794419.733410] lowmem_reserve[]: 0 0 0 0 <4>[4794419.733411] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[4794419.733415] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[4794419.733417] Node 0 Normal: 89201*4kB 9*8kB 10*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 357036kB <4>[4794419.733420] 37938808 total pagecache pages <4>[4794419.733421] 0 pages in swap cache <4>[4794419.733422] Swap cache stats: add 0, delete 0, find 0/0 <4>[4794419.733422] Free swap = 0kB <4>[4794419.733423] Total swap = 0kB <6>[4794420.068556] 67108863 pages RAM <6>[4794420.068557] 1015653 pages reserved <6>[4794420.068557] 36725426 pages shared <6>[4794420.068558] 28901310 pages non-shared <3>[4794420.068621] LustreError: 14890:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[4794420.068623] LustreError: 14890:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1488410120 total bytes allocated by lnet <4>[4794420.386080] Pid: 14888, comm: ptlrpcd_0 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[4794420.395667] Call Trace: <4>[4794420.398817] [] ? __alloc_pages_nodemask+0x7dc/0x950 <4>[4794420.406541] [] ? kiblnd_check_sends+0x2ba/0x610 [ko2iblnd] <4>[4794420.415147] [] ? common_interrupt+0xe/0x13 <4>[4794420.421988] [] ? kmem_getpages+0x62/0x170 <4>[4794420.428732] [] ? fallback_alloc+0x1ba/0x270 <4>[4794420.435670] [] ? cache_grow+0x2cf/0x320 <4>[4794420.442214] [] ? ____cache_alloc_node+0x99/0x160 <4>[4794420.449644] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[4794420.458257] [] ? __kmalloc+0x199/0x230 <4>[4794420.464710] [] ? lprocfs_stats_alloc_one+0x91/0x390 [lvfs] <4>[4794420.473322] [] ? lprocfs_counter_add+0x1a8/0x1c0 [lvfs] <4>[4794420.481448] [] ? ptl_send_rpc+0x4de/0xd30 [ptlrpc] <4>[4794420.489095] [] ? lustre_msg_get_slv+0x8c/0x110 [ptlrpc] <4>[4794420.497225] [] ? ptlrpc_send_new_req+0x454/0x790 [ptlrpc] <4>[4794420.505762] [] ? ptlrpc_check_set+0x9a8/0x1bf0 [ptlrpc] <4>[4794420.513891] [] ? ptlrpcd_check+0x3d3/0x610 [ptlrpc] <4>[4794420.521638] [] ? ptlrpcd+0x185/0x370 [ptlrpc] <4>[4794420.528770] [] ? default_wake_function+0x0/0x20 <4>[4794420.536118] [] ? ptlrpcd+0x0/0x370 [ptlrpc] <4>[4794420.543061] [] ? kthread+0x9e/0xc0 <4>[4794420.549126] [] ? child_rip+0xa/0x20 <4>[4794420.555284] [] ? kthread+0x0/0xc0 <4>[4794420.561251] [] ? child_rip+0x0/0x20 <6>[4794420.567409] Mem-Info: <4>[4794420.570362] Node 0 DMA per-cpu: <4>[4794420.574288] CPU 0: hi: 0, btch: 1 usd: 0 <4>[4794420.580063] CPU 1: hi: 0, btch: 1 usd: 0 <4>[4794420.585833] CPU 2: hi: 0, btch: 1 usd: 0 <4>[4794420.591602] CPU 3: hi: 0, btch: 1 usd: 0 <4>[4794420.597371] CPU 4: hi: 0, btch: 1 usd: 0 <4>[4794420.603143] CPU 5: hi: 0, btch: 1 usd: 0 <4>[4794420.608913] CPU 6: hi: 0, btch: 1 usd: 0 <4>[4794420.614682] CPU 7: hi: 0, btch: 1 usd: 0 <4>[4794420.620452] Node 0 DMA32 per-cpu: <4>[4794420.624574] CPU 0: hi: 186, btch: 31 usd: 0 <4>[4794420.630342] CPU 1: hi: 186, btch: 31 usd: 0 <4>[4794420.636115] CPU 2: hi: 186, btch: 31 usd: 0 <4>[4794420.641886] CPU 3: hi: 186, btch: 31 usd: 0 <4>[4794420.647662] CPU 4: hi: 186, btch: 31 usd: 0 <4>[4794420.653530] CPU 5: hi: 186, btch: 31 usd: 0 <4>[4794420.659301] CPU 6: hi: 186, btch: 31 usd: 0 <4>[4794420.665071] CPU 7: hi: 186, btch: 31 usd: 0 <4>[4794420.670840] Node 0 Normal per-cpu: <4>[4794420.675067] CPU 0: hi: 186, btch: 31 usd: 179 <4>[4794420.680837] CPU 1: hi: 186, btch: 31 usd: 2 <4>[4794420.686608] CPU 2: hi: 186, btch: 31 usd: 175 <4>[4794420.692383] CPU 3: hi: 186, btch: 31 usd: 51 <4>[4794420.698154] CPU 4: hi: 186, btch: 31 usd: 23 <4>[4794420.703927] CPU 5: hi: 186, btch: 31 usd: 19 <4>[4794420.709696] CPU 6: hi: 186, btch: 31 usd: 156 <4>[4794420.715465] CPU 7: hi: 186, btch: 31 usd: 84 <4>[4794420.721235] active_anon:3410342 inactive_anon:157871 isolated_anon:0 <4>[4794420.721235] active_file:8915894 inactive_file:28812830 isolated_file:0 <4>[4794420.721236] unevictable:8901 dirty:16226 writeback:2970 unstable:0 <4>[4794420.721236] free:189022 slab_reclaimable:9433675 slab_unreclaimable:14278451 <4>[4794420.721237] mapped:9211 shmem:214095 pagetables:7995 bounce:0 <4>[4794420.759533] Node 0 DMA free:15740kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15348kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[4794420.801578] lowmem_reserve[]: 0 1880 258420 258420 <4>[4794420.807407] Node 0 DMA32 free:385116kB min:488kB low:608kB high:732kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1925128kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes <4>[4794420.850514] lowmem_reserve[]: 0 0 256540 256540 <4>[4794420.856056] Node 0 Normal free:412436kB min:67088kB low:83860kB high:100632kB active_anon:13641368kB inactive_anon:631484kB active_file:35663576kB inactive_file:115218484kB unevictable:35604kB isolated(anon):0kB isolated(file):0kB present:262696960kB mlocked:0kB dirty:81144kB writeback:11880kB mapped:36844kB shmem:856380kB slab_reclaimable:37722672kB slab_unreclaimable:57102552kB kernel_stack:74624kB pagetables:31980kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no <4>[4794420.907076] lowmem_reserve[]: 0 0 0 0 <4>[4794420.911654] Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB <4>[4794420.929473] Node 0 DMA32: 9*4kB 3*8kB 4*16kB 7*32kB 6*64kB 5*128kB 7*256kB 10*512kB 8*1024kB 4*2048kB 88*4096kB = 385116kB <4>[4794420.942611] Node 0 Normal: 89097*4kB 222*8kB 2418*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 396852kB <4>[4794420.956509] 37939860 total pagecache pages <4>[4794420.961500] 0 pages in swap cache <4>[4794420.965614] Swap cache stats: add 0, delete 0, find 0/0 <4>[4794420.971871] Free swap = 0kB <4>[4794420.975509] Total swap = 0kB <6>[4794421.316469] 67108863 pages RAM <6>[4794421.320295] 1015653 pages reserved <6>[4794421.324509] 36750628 pages shared <6>[4794421.328630] 28891337 pages non-shared <3>[4794421.333256] LustreError: 14888:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: out of memory at /data/buildsystem/jsimmons-atlas/rpmbuild/BUILD/lustre-2.5.5/lustre/lvfs/lvfs_lib.c:157 (tried to alloc '(stats->ls_percpu[cpuid])' = 4224) <3>[4794421.358024] LustreError: 14888:0:(lvfs_lib.c:157:lprocfs_stats_alloc_one()) LNET: 1488343144 total bytes allocated by lnet <4>[4794839.129576] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4794843.152959] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4794845.273781] ACPI: No handler for Region [POWR] (ffff884027a592b8) [IPMI] <4>[4794863.688807] Lustre: atlas1-MDT0000: Client 5a47e11b-80f6-f257-8dde-4cd7bde7a2d0 (at 629@gni3) reconnecting <4>[4794864.190633] Lustre: atlas1-MDT0000: Client 0e890f64-cc33-cb9f-fa60-9f9d5467f8d2 (at 634@gni3) reconnecting <4>[4794864.202071] Lustre: Skipped 2 previous similar messages <4>[4794873.536344] Lustre: Unmounted atlas1-client <4>[4795622.042501] Lustre: atlas1-MDT0000: haven't heard from client cd990e1c-4b59-befb-75a6-342baaae177c (at 383@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f4776f400, cur 1461068706 expire 1461067806 last 1461067354 <4>[4795623.307015] Lustre: atlas1-MDT0000: haven't heard from client 2dd4786d-cf4c-be00-50d2-1f3ae1cb0070 (at 696@gni3) in 1345 seconds. I think it's dead, and I am evicting it. exp ffff883f327d6c00, cur 1461068707 expire 1461067806 last 1461067362 <4>[4795623.332043] Lustre: Skipped 167 previous similar messages <4>[4796073.044215] Lustre: atlas1-MDT0000: haven't heard from client 1b2cd946-265a-46f5-eda5-3760706c2bf5 (at 485@gni3) in 1162 seconds. I think it's dead, and I am evicting it. exp ffff883f3d066c00, cur 1461069157 expire 1461068257 last 1461067995 <4>[4796073.069235] Lustre: Skipped 539 previous similar messages <4>[4796785.985325] Lustre: atlas1-MDT0000: Client 9a7e9e88-50ec-3d0a-9182-2d2672622596 (at 10942@gni100) reconnecting <4>[4797963.040819] Lustre: atlas1-MDT0000: haven't heard from client 8f806322-7c34-9991-e929-b67cc0c90ffa (at 282@gni3) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff883f3309c400, cur 1461071047 expire 1461070147 last 1461069695 <4>[4797965.571074] Lustre: atlas1-MDT0000: haven't heard from client bc56134d-3299-3609-b87a-2d57e3b694ec (at 381@gni3) in 1346 seconds. I think it's dead, and I am evicting it. exp ffff883f30ed2800, cur 1461071049 expire 1461070147 last 1461069703 <4>[4797965.596166] Lustre: Skipped 24 previous similar messages <4>[4801313.036357] Lustre: atlas1-MDT0000: haven't heard from client fc7fe7d9-c4e3-40fa-8b74-5d90dc1aca86 (at 15895@gni100) in 1352 seconds. I think it's dead, and I am evicting it. exp ffff8837325b3400, cur 1461074397 expire 1461073497 last 1461073045 <4>[4801313.061773] Lustre: Skipped 2 previous similar messages <0>[4805210.570311] LustreError: 15316:0:(mdt_handler.c:3224:mdt_lock_handle_fini()) ASSERTION( !lustre_handle_is_used(&lh->mlh_reg_lh) ) failed: <0>[4805210.570478] LustreError: 15659:0:(mdt_handler.c:3224:mdt_lock_handle_fini()) ASSERTION( !lustre_handle_is_used(&lh->mlh_reg_lh) ) failed: <0>[4805210.570480] LustreError: 15659:0:(mdt_handler.c:3224:mdt_lock_handle_fini()) LBUG <4>[4805210.570481] Pid: 15659, comm: mdt00_441 <4>[4805210.570481] <4>[4805210.570482] Call Trace: <4>[4805210.570495] [] libcfs_debug_dumpstack+0x55/0x80 [libcfs] <4>[4805210.570499] [] lbug_with_loc+0x47/0xb0 [libcfs] <4>[4805210.570511] [] mdt_lock_handle_fini+0x4b/0x80 [mdt] <4>[4805210.570517] [] mdt_thread_info_fini+0xe0/0x190 [mdt] <4>[4805210.570523] [] mdt_handle_common+0x653/0x1470 [mdt] <4>[4805210.570533] [] mds_regular_handle+0x15/0x20 [mdt] <4>[4805210.570563] [] ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[4805210.570581] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[4805210.570584] [] kthread+0x9e/0xc0 <4>[4805210.570587] [] child_rip+0xa/0x20 <4>[4805210.570588] [] ? kthread+0x0/0xc0 <4>[4805210.570589] [] ? child_rip+0x0/0x20 <4>[4805210.570590] <0>[4805210.571042] Kernel panic - not syncing: LBUG <4>[4805210.571044] Pid: 15659, comm: mdt00_441 Not tainted 2.6.32-573.12.1.el6.atlas.x86_64 #1 <4>[4805210.571045] Call Trace: <4>[4805210.571050] [] ? panic+0xa7/0x16f <4>[4805210.571055] [] ? lbug_with_loc+0x9b/0xb0 [libcfs] <4>[4805210.571061] [] ? mdt_lock_handle_fini+0x4b/0x80 [mdt] <4>[4805210.571067] [] ? mdt_thread_info_fini+0xe0/0x190 [mdt] <4>[4805210.571072] [] ? mdt_handle_common+0x653/0x1470 [mdt] <4>[4805210.571080] [] ? mds_regular_handle+0x15/0x20 [mdt] <4>[4805210.571097] [] ? ptlrpc_main+0xf8e/0x1af0 [ptlrpc] <4>[4805210.571115] [] ? ptlrpc_main+0x0/0x1af0 [ptlrpc] <4>[4805210.571117] [] ? kthread+0x9e/0xc0 <4>[4805210.571119] [] ? child_rip+0xa/0x20 <4>[4805210.571120] [] ? kthread+0x0/0xc0 <4>[4805210.571122] [] ? child_rip+0x0/0x20