[ 0.000000] microcode: microcode updated early to revision 0x710, date = 2013-06-17 [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.10.0-693.5.2.el7_lustre.x86_64 (jenkins@trevis-310-el7-x8664-2.trevis.hpdd.intel.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon Nov 27 15:30:51 UTC 2017 [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.5.2.el7_lustre.x86_64 root=UUID=2fbe2fc1-ea16-409a-8e17-89fecbe37191 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000008d3ff] usable [ 0.000000] BIOS-e820: [mem 0x000000000008d400-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bb3c6fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bb3c7000-0x00000000bdd2efff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000bdd2f000-0x00000000bddccfff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bddcd000-0x00000000bdea0fff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000bdea1000-0x00000000bdf2efff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x00000000bdf2f000-0x00000000bdfabfff] ACPI data [ 0.000000] BIOS-e820: [mem 0x00000000bdfac000-0x00000000bdffffff] usable [ 0.000000] BIOS-e820: [mem 0x00000000be000000-0x00000000cfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed19000-0x00000000fed19fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fee00000-0x00000000fee00fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ffa20000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000083fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.6 present. [ 0.000000] DMI: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x840000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-BFFFF uncachable [ 0.000000] C0000-DBFFF write-through [ 0.000000] DC000-DFFFF write-protect [ 0.000000] E0000-E7FFF uncachable [ 0.000000] E8000-FFFFF write-protect [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 000000000000 mask 3FFF80000000 write-back [ 0.000000] 1 base 000080000000 mask 3FFFC0000000 write-back [ 0.000000] 2 base 000100000000 mask 3FFF00000000 write-back [ 0.000000] 3 base 000200000000 mask 3FFE00000000 write-back [ 0.000000] 4 base 000400000000 mask 3FFC00000000 write-back [ 0.000000] 5 base 000800000000 mask 3FFFC0000000 write-back [ 0.000000] 6 base 0000FF800000 mask 3FFFFF800000 write-protect [ 0.000000] 7 disabled [ 0.000000] 8 disabled [ 0.000000] 9 disabled [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106 [ 0.000000] e820: last_pfn = 0xbe000 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fcd80-0x000fcd8f] mapped at [ffff8800000fcd80] [ 0.000000] Base memory trampoline at [ffff880000087000] 87000 size 24576 [ 0.000000] Using GB pages for direct mapping [ 0.000000] BRK [0x01fe9000, 0x01fe9fff] PGTABLE [ 0.000000] BRK [0x01fea000, 0x01feafff] PGTABLE [ 0.000000] BRK [0x01feb000, 0x01febfff] PGTABLE [ 0.000000] BRK [0x01fec000, 0x01fecfff] PGTABLE [ 0.000000] RAMDISK: [mem 0x3596a000-0x36cacfff] [ 0.000000] Early table checksum verification disabled [ 0.000000] ACPI: RSDP 00000000000f0410 00024 (v02 INTEL ) [ 0.000000] ACPI: XSDT 00000000bdfa9d98 000BC (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: FACP 00000000bdfa9918 000F4 (v04 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20130517/tbfadt-672) [ 0.000000] ACPI: DSDT 00000000bdf8f018 18762 (v02 INTEL S2600GZ 00000008 INTL 20100331) [ 0.000000] ACPI: FACS 00000000bdfa9f40 00040 [ 0.000000] ACPI: APIC 00000000bdfa8018 00BAA (v03 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SPMI 00000000bdfabb18 00040 (v05 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: MCFG 00000000bdfaba98 0003C (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: WDDT 00000000bdfabf18 00040 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SRAT 00000000bdf8cc18 002A8 (v03 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SLIT 00000000bdfabe98 00030 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: MSCT 00000000bdfaae18 00090 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: HPET 00000000bdfabe18 00038 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: SSDT 00000000bdfabd18 0002B (v02 INTEL S2600GZ 00001000 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bddcd018 D30C8 (v02 INTEL S2600GZ 00004000 INTL 20100331) [ 0.000000] ACPI: SPCR 00000000bdfabd98 00050 (v01 INTEL S2600GZ 06222004 INTL 20090903) [ 0.000000] ACPI: HEST 00000000bdf8ef18 000A8 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: BERT 00000000bdfabc18 00030 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: ERST 00000000bdf8ec98 00230 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: EINJ 00000000bdfa9798 00130 (v01 INTEL S2600GZ 00000001 INTL 00000001) [ 0.000000] ACPI: SSDT 00000000bdf8a018 01729 (v02 INTEL S2600GZ 00000002 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bdfabc98 00045 (v02 INTEL S2600GZ 00000001 INTL 20100331) [ 0.000000] ACPI: SSDT 00000000bdf8de18 00181 (v02 INTEL S2600GZ 00000003 INTL 20100331) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x04 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x05 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x06 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x07 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x08 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x09 -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0a -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0b -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0c -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0d -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0e -> Node 0 [ 0.000000] SRAT: PXM 0 -> APIC 0x0f -> Node 0 [ 0.000000] SRAT: PXM 1 -> APIC 0x20 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x21 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x22 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x23 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x24 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x25 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x26 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x27 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x28 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x29 -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2a -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2b -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2c -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2d -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2e -> Node 1 [ 0.000000] SRAT: PXM 1 -> APIC 0x2f -> Node 1 [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0xbfffffff] [ 0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x43fffffff] [ 0.000000] SRAT: Node 1 PXM 1 [mem 0x440000000-0x83fffffff] [ 0.000000] NUMA: Initialized distance table, cnt=2 [ 0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x43fffffff] -> [mem 0x00000000-0x43fffffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x43ffd9000-0x43fffffff] [ 0.000000] NODE_DATA(1) allocated [mem 0x83ffd6000-0x83fffcfff] [ 0.000000] Reserving 320MB of memory at 528MB for crashkernel (System RAM: 32691MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00001000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0xffffffff] [ 0.000000] Normal [mem 0x100000000-0x83fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00001000-0x0008cfff] [ 0.000000] node 0: [mem 0x00100000-0xbb3c6fff] [ 0.000000] node 0: [mem 0xbdfac000-0xbdffffff] [ 0.000000] node 0: [mem 0x100000000-0x43fffffff] [ 0.000000] node 1: [mem 0x440000000-0x83fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00001000-0x43fffffff] [ 0.000000] On node 0 totalpages: 4174759 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 21 pages reserved [ 0.000000] DMA zone: 3980 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 11921 pages used for memmap [ 0.000000] DMA32 zone: 762907 pages, LIFO batch:31 [ 0.000000] Normal zone: 53248 pages used for memmap [ 0.000000] Normal zone: 3407872 pages, LIFO batch:31 [ 0.000000] Initmem setup node 1 [mem 0x440000000-0x83fffffff] [ 0.000000] On node 1 totalpages: 4194304 [ 0.000000] Normal zone: 65536 pages used for memmap [ 0.000000] Normal zone: 4194304 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x02] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x04] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x06] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x08] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x05] lapic_id[0x0a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x06] lapic_id[0x0c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x07] lapic_id[0x0e] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x08] lapic_id[0x20] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x09] lapic_id[0x22] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x24] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x26] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x28] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2a] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2c] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x01] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x03] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x05] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x07] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x14] lapic_id[0x09] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x15] lapic_id[0x0b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x16] lapic_id[0x0d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x17] lapic_id[0x0f] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x18] lapic_id[0x21] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x19] lapic_id[0x23] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x25] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x27] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x29] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x2b] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x2d] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x2f] enabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x20] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x21] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x22] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x23] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x24] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x25] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x26] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x27] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x28] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x29] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x2f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x30] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x31] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x32] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x33] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x34] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x35] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x36] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x37] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x38] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x39] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x3f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x40] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x41] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x42] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x43] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x44] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x45] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x46] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x47] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x48] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x49] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x4f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x50] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x51] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x52] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x53] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x54] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x55] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x56] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x57] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x58] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x59] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x5f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x60] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x61] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x62] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x63] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x64] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x65] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x66] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x67] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x68] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x69] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6a] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6b] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6c] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6d] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6e] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x6f] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x70] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x71] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x72] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x73] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x74] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x75] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x76] lapic_id[0xff] disabled) [ 0.000000] ACPI: LAPIC (acpi_id[0x77] lapic_id[0xff] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x00] uid[0x00] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x01] uid[0x01] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x02] uid[0x02] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x03] uid[0x03] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x04] uid[0x04] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x05] uid[0x05] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x06] uid[0x06] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x07] uid[0x07] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x08] uid[0x08] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x09] uid[0x09] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0a] uid[0x0a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0b] uid[0x0b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0c] uid[0x0c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0d] uid[0x0d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0e] uid[0x0e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x0f] uid[0x0f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x10] uid[0x10] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x11] uid[0x11] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x12] uid[0x12] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x13] uid[0x13] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x14] uid[0x14] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x15] uid[0x15] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x16] uid[0x16] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x17] uid[0x17] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x18] uid[0x18] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x19] uid[0x19] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1a] uid[0x1a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1b] uid[0x1b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1c] uid[0x1c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1d] uid[0x1d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1e] uid[0x1e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x1f] uid[0x1f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x20] uid[0x20] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x21] uid[0x21] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x22] uid[0x22] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x23] uid[0x23] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x24] uid[0x24] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x25] uid[0x25] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x26] uid[0x26] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x27] uid[0x27] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x28] uid[0x28] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x29] uid[0x29] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2a] uid[0x2a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2b] uid[0x2b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2c] uid[0x2c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2d] uid[0x2d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2e] uid[0x2e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x2f] uid[0x2f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x30] uid[0x30] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x31] uid[0x31] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x32] uid[0x32] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x33] uid[0x33] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x34] uid[0x34] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x35] uid[0x35] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x36] uid[0x36] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x37] uid[0x37] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x38] uid[0x38] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x39] uid[0x39] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3a] uid[0x3a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3b] uid[0x3b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3c] uid[0x3c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3d] uid[0x3d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3e] uid[0x3e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x3f] uid[0x3f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x40] uid[0x40] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x41] uid[0x41] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x42] uid[0x42] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x43] uid[0x43] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x44] uid[0x44] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x45] uid[0x45] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x46] uid[0x46] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x47] uid[0x47] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x48] uid[0x48] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x49] uid[0x49] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4a] uid[0x4a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4b] uid[0x4b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4c] uid[0x4c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4d] uid[0x4d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4e] uid[0x4e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x4f] uid[0x4f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x50] uid[0x50] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x51] uid[0x51] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x52] uid[0x52] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x53] uid[0x53] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x54] uid[0x54] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x55] uid[0x55] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x56] uid[0x56] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x57] uid[0x57] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x58] uid[0x58] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x59] uid[0x59] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5a] uid[0x5a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5b] uid[0x5b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5c] uid[0x5c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5d] uid[0x5d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5e] uid[0x5e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x5f] uid[0x5f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x60] uid[0x60] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x61] uid[0x61] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x62] uid[0x62] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x63] uid[0x63] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x64] uid[0x64] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x65] uid[0x65] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x66] uid[0x66] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x67] uid[0x67] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x68] uid[0x68] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x69] uid[0x69] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6a] uid[0x6a] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6b] uid[0x6b] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6c] uid[0x6c] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6d] uid[0x6d] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6e] uid[0x6e] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x6f] uid[0x6f] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x70] uid[0x70] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x71] uid[0x71] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x72] uid[0x72] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x73] uid[0x73] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x74] uid[0x74] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x75] uid[0x75] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x76] uid[0x76] disabled) [ 0.000000] ACPI: X2APIC (apic_id[0x77] uid[0x77] disabled) [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) [ 0.000000] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: IOAPIC (id[0x01] address[0xfec3f000] gsi_base[24]) [ 0.000000] IOAPIC[1]: apic_id 1, version 32, address 0xfec3f000, GSI 24-47 [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec7f000] gsi_base[48]) [ 0.000000] IOAPIC[2]: apic_id 2, version 32, address 0xfec7f000, GSI 48-71 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a701 base: 0xfed00000 [ 0.000000] smpboot: Allowing 240 CPUs, 208 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x0008d000-0x0008dfff] [ 0.000000] PM: Registered nosave memory: [mem 0x0008e000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbb3c7000-0xbdd2efff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdd2f000-0xbddccfff] [ 0.000000] PM: Registered nosave memory: [mem 0xbddcd000-0xbdea0fff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdea1000-0xbdf2efff] [ 0.000000] PM: Registered nosave memory: [mem 0xbdf2f000-0xbdfabfff] [ 0.000000] PM: Registered nosave memory: [mem 0xbe000000-0xcfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xd0000000-0xfebfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec00000-0xfec00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfec01000-0xfed18fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed19000-0xfed19fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1a000-0xfed1bfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed1c000-0xfed1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfed20000-0xfedfffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee00000-0xfee00fff] [ 0.000000] PM: Registered nosave memory: [mem 0xfee01000-0xffa1ffff] [ 0.000000] PM: Registered nosave memory: [mem 0xffa20000-0xffffffff] [ 0.000000] e820: [mem 0xd0000000-0xfebfffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:240 nr_cpu_ids:240 nr_node_ids:2 [ 0.000000] PERCPU: Embedded 33 pages/cpu @ffff88042e000000 s97048 r8192 d29928 u262144 [ 0.000000] pcpu-alloc: s97048 r8192 d29928 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 000 001 002 003 004 005 006 007 [ 0.000000] pcpu-alloc: [0] 016 017 018 019 020 021 022 023 [ 0.000000] pcpu-alloc: [0] 032 034 036 038 040 042 044 046 [ 0.000000] pcpu-alloc: [0] 048 050 052 054 056 058 060 062 [ 0.000000] pcpu-alloc: [0] 064 066 068 070 072 074 076 078 [ 0.000000] pcpu-alloc: [0] 080 082 084 086 088 090 092 094 [ 0.000000] pcpu-alloc: [0] 096 098 100 102 104 106 108 110 [ 0.000000] pcpu-alloc: [0] 112 114 116 118 120 122 124 126 [ 0.000000] pcpu-alloc: [0] 128 130 132 134 136 138 140 142 [ 0.000000] pcpu-alloc: [0] 144 146 148 150 152 154 156 158 [ 0.000000] pcpu-alloc: [0] 160 162 164 166 168 170 172 174 [ 0.000000] pcpu-alloc: [0] 176 178 180 182 184 186 188 190 [ 0.000000] pcpu-alloc: [0] 192 194 196 198 200 202 204 206 [ 0.000000] pcpu-alloc: [0] 208 210 212 214 216 218 220 222 [ 0.000000] pcpu-alloc: [0] 224 226 228 230 232 234 236 238 [ 0.000000] pcpu-alloc: [1] 008 009 010 011 012 013 014 015 [ 0.000000] pcpu-alloc: [1] 024 025 026 027 028 029 030 031 [ 0.000000] pcpu-alloc: [1] 033 035 037 039 041 043 045 047 [ 0.000000] pcpu-alloc: [1] 049 051 053 055 057 059 061 063 [ 0.000000] pcpu-alloc: [1] 065 067 069 071 073 075 077 079 [ 0.000000] pcpu-alloc: [1] 081 083 085 087 089 091 093 095 [ 0.000000] pcpu-alloc: [1] 097 099 101 103 105 107 109 111 [ 0.000000] pcpu-alloc: [1] 113 115 117 119 121 123 125 127 [ 0.000000] pcpu-alloc: [1] 129 131 133 135 137 139 141 143 [ 0.000000] pcpu-alloc: [1] 145 147 149 151 153 155 157 159 [ 0.000000] pcpu-alloc: [1] 161 163 165 167 169 171 173 175 [ 0.000000] pcpu-alloc: [1] 177 179 181 183 185 187 189 191 [ 0.000000] pcpu-alloc: [1] 193 195 197 199 201 203 205 207 [ 0.000000] pcpu-alloc: [1] 209 211 213 215 217 219 221 223 [ 0.000000] pcpu-alloc: [1] 225 227 229 231 233 235 237 239 [ 0.000000] Built 2 zonelists in Zone order, mobility grouping on. Total pages: 8238273 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.10.0-693.5.2.el7_lustre.x86_64 root=UUID=2fbe2fc1-ea16-409a-8e17-89fecbe37191 ro crashkernel=320M console=tty0 LANG=en_US.UTF-8 console=ttyS0,115200 net.ifnames=0 [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100 [ 0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form [ 0.000000] Memory: 6832456k/34603008k available (6886k kernel code, 1126756k absent, 986540k reserved, 4545k data, 1764k init) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=240, Nodes=2 [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=240. [ 0.000000] NR_IRQS:327936 nr_irqs:3160 0 [ 0.000000] Console: colour VGA+ 80x25 [ 0.000000] console [tty0] enabled [ 0.000000] console [ttyS0] enabled [ 0.000000] allocated 134217728 bytes of page_cgroup [ 0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups [ 0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2693.422 MHz processor [ 0.000101] Calibrating delay loop (skipped), value calculated using timer frequency.. 5386.84 BogoMIPS (lpj=2693422) [ 0.012531] pid_max: default: 245760 minimum: 1920 [ 0.018821] Security Framework initialized [ 0.023685] SELinux: Initializing. [ 0.027932] SELinux: Starting in permissive mode [ 0.027933] Yama: becoming mindful. [ 0.034572] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes) [ 0.049888] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes) [ 0.061290] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.069295] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.079717] Initializing cgroup subsys memory [ 0.084912] Initializing cgroup subsys devices [ 0.090155] Initializing cgroup subsys freezer [ 0.095399] Initializing cgroup subsys net_cls [ 0.100642] Initializing cgroup subsys blkio [ 0.105855] Initializing cgroup subsys perf_event [ 0.111431] Initializing cgroup subsys hugetlb [ 0.116677] Initializing cgroup subsys pids [ 0.121626] Initializing cgroup subsys net_prio [ 0.127120] CPU: Physical Processor ID: 0 [ 0.131877] CPU: Processor Core ID: 0 [ 0.136248] mce: CPU supports 20 MCE banks [ 0.141114] CPU0: Thermal monitoring enabled (TM1) [ 0.146790] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0 [ 0.153201] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32 [ 0.159804] tlb_flushall_shift: 6 [ 0.163933] Freeing SMP alternatives: 24k freed [ 0.171792] ACPI: Core revision 20130517 [ 0.213686] ACPI: All ACPI Tables successfully acquired [ 0.234845] ftrace: allocating 26589 entries in 104 pages [ 0.253841] smpboot: Max logical packages: 30 [ 0.259613] IRQ remapping doesn't support X2APIC mode, disable x2apic. [ 0.267263] Switched APIC routing to physical flat. [ 0.273601] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.290581] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz (fam: 06, model: 2d, stepping: 07) [ 0.301705] TSC deadline timer enabled [ 0.301738] Performance Events: PEBS fmt1+, 16-deep LBR, SandyBridge events, full-width counters, Intel PMU driver. [ 0.314284] ... version: 3 [ 0.319039] ... bit width: 48 [ 0.323888] ... generic registers: 4 [ 0.328638] ... value mask: 0000ffffffffffff [ 0.334849] ... max period: 00007fffffffffff [ 0.341058] ... fixed-purpose events: 3 [ 0.345806] ... event mask: 000000070000000f [ 0.377334] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. [ 0.361453] smpboot: Booting Node 0, Processors #1 #2 #3 #4 #5 #6 #7 OK [ 0.414498] smpboot: Booting Node 1, Processors #8 #9 #10 #11 #12 #13 #14 #15 OK [ 0.541483] smpboot: Booting Node 0, Processors #16 #17 #18 #19 #20 #21 #22 #23 OK [ 0.576066] smpboot: Booting Node 1, Processors #24 #25 #26 #27 #28 #29 #30 #31 [ 0.616533] Brought up 32 CPUs [ 0.620694] smpboot: Total of 32 processors activated (172459.32 BogoMIPS) [ 0.745248] node 0 initialised, 2813971 pages in 62ms [ 0.762584] node 1 initialised, 3600339 pages in 74ms [ 0.769793] devtmpfs: initialized [ 0.782777] EVM: security.selinux [ 0.786768] EVM: security.ima [ 0.790368] EVM: security.capability [ 0.794924] PM: Registering ACPI NVS region [mem 0xbdd2f000-0xbddccfff] (647168 bytes) [ 0.804297] PM: Registering ACPI NVS region [mem 0xbdea1000-0xbdf2efff] (581632 bytes) [ 0.816164] atomic64 test passed for x86-64 platform with CX8 and with SSE [ 0.824145] pinctrl core: initialized pinctrl subsystem [ 0.830548] RTC time: 0:08:40, date: 12/14/17 [ 0.836307] NET: Registered protocol family 16 [ 0.842002] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 0.850961] ACPI: bus type PCI registered [ 0.855729] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.863844] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xc0000000-0xcfffffff] (base 0xc0000000) [ 0.874750] PCI: MMCONFIG at [mem 0xc0000000-0xcfffffff] reserved in E820 [ 0.882675] PCI: Using configuration type 1 for base access [ 0.889252] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.907542] ACPI: Added _OSI(Module Device) [ 0.912509] ACPI: Added _OSI(Processor Device) [ 0.917761] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.923308] ACPI: Added _OSI(Processor Aggregator Device) [ 0.947154] ACPI: EC: Look up EC in DSDT [ 0.964997] ACPI: Executed 1 blocks of module-level executable AML code [ 1.236788] [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored [ 1.267513] ACPI: Interpreter enabled [ 1.271916] ACPI: (supports S0 S1 S5) [ 1.276292] ACPI: Using IOAPIC for interrupt routing [ 1.282242] HEST: Table parsing has been initialized. [ 1.288178] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 1.351563] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7e]) [ 1.358771] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.368904] acpi PNP0A08:00: _OSC: platform does not support [AER] [ 1.376487] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 1.385548] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.395445] acpi PNP0A08:00: ignoring host bridge window [mem 0x000d0000-0x000d3fff window] (conflicts with Adapter ROM [mem 0x000c8000-0x000d1bff]) [ 1.410823] acpi PNP0A08:00: ignoring host bridge window [mem 0x000d8000-0x000dbfff window] (conflicts with Adapter ROM [mem 0x000d2000-0x000d8bff]) [ 1.426560] PCI host bridge to bus 0000:00 [ 1.431429] pci_bus 0000:00: root bus resource [bus 00-7e] [ 1.437851] pci_bus 0000:00: root bus resource [io 0x0000-0xbfff window] [ 1.445728] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.454589] pci_bus 0000:00: root bus resource [mem 0x000c0000-0x000c3fff window] [ 1.463452] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window] [ 1.472308] pci_bus 0000:00: root bus resource [mem 0x000c8000-0x000cbfff window] [ 1.481165] pci_bus 0000:00: root bus resource [mem 0x000cc000-0x000cffff window] [ 1.490022] pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff window] [ 1.498877] pci_bus 0000:00: root bus resource [mem 0x000dc000-0x000dffff window] [ 1.507737] pci_bus 0000:00: root bus resource [mem 0x000e0000-0x000e3fff window] [ 1.516597] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000e7fff window] [ 1.525459] pci_bus 0000:00: root bus resource [mem 0x000e8000-0x000ebfff window] [ 1.534319] pci_bus 0000:00: root bus resource [mem 0x000ec000-0x000effff window] [ 1.543178] pci_bus 0000:00: root bus resource [mem 0x000f0000-0x000fffff window] [ 1.552034] pci_bus 0000:00: root bus resource [mem 0xd0000000-0xebffffff window] [ 1.560895] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 1.570548] pci 0000:00:00.0: [8086:3c00] type 00 class 0x060000 [ 1.570645] pci 0000:00:00.0: PME# supported from D0 D3hot D3cold [ 1.570794] pci 0000:00:01.0: [8086:3c02] type 01 class 0x060400 [ 1.570895] pci 0000:00:01.0: PME# supported from D0 D3hot D3cold [ 1.570995] pci 0000:00:01.0: System wakeup disabled by ACPI [ 1.577659] pci 0000:00:01.1: [8086:3c03] type 01 class 0x060400 [ 1.577759] pci 0000:00:01.1: PME# supported from D0 D3hot D3cold [ 1.577857] pci 0000:00:01.1: System wakeup disabled by ACPI [ 1.584536] pci 0000:00:02.0: [8086:3c04] type 01 class 0x060400 [ 1.584639] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold [ 1.584744] pci 0000:00:02.0: System wakeup disabled by ACPI [ 1.591414] pci 0000:00:02.2: [8086:3c06] type 01 class 0x060400 [ 1.591515] pci 0000:00:02.2: PME# supported from D0 D3hot D3cold [ 1.591623] pci 0000:00:02.2: System wakeup disabled by ACPI [ 1.598304] pci 0000:00:03.0: [8086:3c08] type 01 class 0x060400 [ 1.598405] pci 0000:00:03.0: PME# supported from D0 D3hot D3cold [ 1.598502] pci 0000:00:03.0: System wakeup disabled by ACPI [ 1.605175] pci 0000:00:04.0: [8086:3c20] type 00 class 0x088000 [ 1.605199] pci 0000:00:04.0: reg 0x10: [mem 0xebf90000-0xebf93fff 64bit] [ 1.605401] pci 0000:00:04.1: [8086:3c21] type 00 class 0x088000 [ 1.605424] pci 0000:00:04.1: reg 0x10: [mem 0xebf80000-0xebf83fff 64bit] [ 1.605622] pci 0000:00:04.2: [8086:3c22] type 00 class 0x088000 [ 1.605644] pci 0000:00:04.2: reg 0x10: [mem 0xebf70000-0xebf73fff 64bit] [ 1.605841] pci 0000:00:04.3: [8086:3c23] type 00 class 0x088000 [ 1.605863] pci 0000:00:04.3: reg 0x10: [mem 0xebf60000-0xebf63fff 64bit] [ 1.606072] pci 0000:00:04.4: [8086:3c24] type 00 class 0x088000 [ 1.606093] pci 0000:00:04.4: reg 0x10: [mem 0xebf50000-0xebf53fff 64bit] [ 1.606299] pci 0000:00:04.5: [8086:3c25] type 00 class 0x088000 [ 1.606320] pci 0000:00:04.5: reg 0x10: [mem 0xebf40000-0xebf43fff 64bit] [ 1.606511] pci 0000:00:04.6: [8086:3c26] type 00 class 0x088000 [ 1.606533] pci 0000:00:04.6: reg 0x10: [mem 0xebf30000-0xebf33fff 64bit] [ 1.606731] pci 0000:00:04.7: [8086:3c27] type 00 class 0x088000 [ 1.606753] pci 0000:00:04.7: reg 0x10: [mem 0xebf20000-0xebf23fff 64bit] [ 1.606947] pci 0000:00:05.0: [8086:3c28] type 00 class 0x088000 [ 1.607137] pci 0000:00:05.2: [8086:3c2a] type 00 class 0x088000 [ 1.607321] pci 0000:00:05.4: [8086:3c2c] type 00 class 0x080020 [ 1.607338] pci 0000:00:05.4: reg 0x10: [mem 0xd0f60000-0xd0f60fff] [ 1.607541] pci 0000:00:11.0: [8086:1d3e] type 01 class 0x060400 [ 1.607661] pci 0000:00:11.0: PME# supported from D0 D3hot D3cold [ 1.607815] pci 0000:00:16.0: [8086:1d3a] type 00 class 0x078000 [ 1.607840] pci 0000:00:16.0: reg 0x10: [mem 0xd0f50000-0xd0f5000f 64bit] [ 1.607928] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 1.608067] pci 0000:00:16.1: [8086:1d3b] type 00 class 0x078000 [ 1.608093] pci 0000:00:16.1: reg 0x10: [mem 0xd0f40000-0xd0f4000f 64bit] [ 1.608184] pci 0000:00:16.1: PME# supported from D0 D3hot D3cold [ 1.608326] pci 0000:00:1a.0: [8086:1d2d] type 00 class 0x0c0320 [ 1.608350] pci 0000:00:1a.0: reg 0x10: [mem 0xd0f20000-0xd0f203ff] [ 1.608460] pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold [ 1.608567] pci 0000:00:1a.0: System wakeup disabled by ACPI [ 1.615253] pci 0000:00:1c.0: [8086:1d10] type 01 class 0x060400 [ 1.615387] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold [ 1.615480] pci 0000:00:1c.0: System wakeup disabled by ACPI [ 1.622176] pci 0000:00:1c.7: [8086:1d1e] type 01 class 0x060400 [ 1.622279] pci 0000:00:1c.7: PME# supported from D0 D3hot D3cold [ 1.622371] pci 0000:00:1c.7: System wakeup disabled by ACPI [ 1.629067] pci 0000:00:1d.0: [8086:1d26] type 00 class 0x0c0320 [ 1.629091] pci 0000:00:1d.0: reg 0x10: [mem 0xd0f10000-0xd0f103ff] [ 1.629202] pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold [ 1.629307] pci 0000:00:1d.0: System wakeup disabled by ACPI [ 1.635976] pci 0000:00:1e.0: [8086:244e] type 01 class 0x060401 [ 1.636118] pci 0000:00:1e.0: System wakeup disabled by ACPI [ 1.642808] pci 0000:00:1f.0: [8086:1d41] type 00 class 0x060100 [ 1.643057] pci 0000:00:1f.2: [8086:1d02] type 00 class 0x010601 [ 1.643080] pci 0000:00:1f.2: reg 0x10: [io 0x4070-0x4077] [ 1.643091] pci 0000:00:1f.2: reg 0x14: [io 0x4060-0x4063] [ 1.643103] pci 0000:00:1f.2: reg 0x18: [io 0x4050-0x4057] [ 1.643114] pci 0000:00:1f.2: reg 0x1c: [io 0x4040-0x4043] [ 1.643126] pci 0000:00:1f.2: reg 0x20: [io 0x4020-0x403f] [ 1.643137] pci 0000:00:1f.2: reg 0x24: [mem 0xd0f00000-0xd0f007ff] [ 1.643197] pci 0000:00:1f.2: PME# supported from D3hot [ 1.643324] pci 0000:00:1f.3: [8086:1d22] type 00 class 0x0c0500 [ 1.643345] pci 0000:00:1f.3: reg 0x10: [mem 0xebf10000-0xebf100ff 64bit] [ 1.643374] pci 0000:00:1f.3: reg 0x20: [io 0x4000-0x401f] [ 1.643591] pci 0000:00:01.0: PCI bridge to [bus 01] [ 1.649601] acpiphp: Slot [2] registered [ 1.654328] pci 0000:02:00.0: [8086:1521] type 00 class 0x020000 [ 1.654347] pci 0000:02:00.0: reg 0x10: [mem 0xd0b60000-0xd0b7ffff] [ 1.654366] pci 0000:02:00.0: reg 0x18: [io 0x3060-0x307f] [ 1.654377] pci 0000:02:00.0: reg 0x1c: [mem 0xd0bb0000-0xd0bb3fff] [ 1.654495] pci 0000:02:00.0: PME# supported from D0 D3hot D3cold [ 1.654528] pci 0000:02:00.0: reg 0x184: [mem 0xd0ca0000-0xd0ca3fff] [ 1.654533] pci 0000:02:00.0: VF(n) BAR0 space: [mem 0xd0ca0000-0xd0cbffff] (contains BAR0 for 8 VFs) [ 1.665363] pci 0000:02:00.0: reg 0x190: [mem 0xd0c80000-0xd0c83fff] [ 1.665367] pci 0000:02:00.0: VF(n) BAR3 space: [mem 0xd0c80000-0xd0c9ffff] (contains BAR3 for 8 VFs) [ 1.676322] pci 0000:02:00.1: [8086:1521] type 00 class 0x020000 [ 1.676341] pci 0000:02:00.1: reg 0x10: [mem 0xd0b40000-0xd0b5ffff] [ 1.676360] pci 0000:02:00.1: reg 0x18: [io 0x3040-0x305f] [ 1.676371] pci 0000:02:00.1: reg 0x1c: [mem 0xd0ba0000-0xd0ba3fff] [ 1.676482] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold [ 1.676509] pci 0000:02:00.1: reg 0x184: [mem 0xd0c60000-0xd0c63fff] [ 1.676512] pci 0000:02:00.1: VF(n) BAR0 space: [mem 0xd0c60000-0xd0c7ffff] (contains BAR0 for 8 VFs) [ 1.687339] pci 0000:02:00.1: reg 0x190: [mem 0xd0c40000-0xd0c43fff] [ 1.687343] pci 0000:02:00.1: VF(n) BAR3 space: [mem 0xd0c40000-0xd0c5ffff] (contains BAR3 for 8 VFs) [ 1.698286] pci 0000:02:00.2: [8086:1521] type 00 class 0x020000 [ 1.698304] pci 0000:02:00.2: reg 0x10: [mem 0xd0b20000-0xd0b3ffff] [ 1.698323] pci 0000:02:00.2: reg 0x18: [io 0x3020-0x303f] [ 1.698334] pci 0000:02:00.2: reg 0x1c: [mem 0xd0b90000-0xd0b93fff] [ 1.698444] pci 0000:02:00.2: PME# supported from D0 D3hot D3cold [ 1.698470] pci 0000:02:00.2: reg 0x184: [mem 0xd0c20000-0xd0c23fff] [ 1.698474] pci 0000:02:00.2: VF(n) BAR0 space: [mem 0xd0c20000-0xd0c3ffff] (contains BAR0 for 8 VFs) [ 1.709301] pci 0000:02:00.2: reg 0x190: [mem 0xd0c00000-0xd0c03fff] [ 1.709305] pci 0000:02:00.2: VF(n) BAR3 space: [mem 0xd0c00000-0xd0c1ffff] (contains BAR3 for 8 VFs) [ 1.720246] pci 0000:02:00.3: [8086:1521] type 00 class 0x020000 [ 1.720264] pci 0000:02:00.3: reg 0x10: [mem 0xd0b00000-0xd0b1ffff] [ 1.720282] pci 0000:02:00.3: reg 0x18: [io 0x3000-0x301f] [ 1.720294] pci 0000:02:00.3: reg 0x1c: [mem 0xd0b80000-0xd0b83fff] [ 1.720403] pci 0000:02:00.3: PME# supported from D0 D3hot D3cold [ 1.720430] pci 0000:02:00.3: reg 0x184: [mem 0xd0be0000-0xd0be3fff] [ 1.720434] pci 0000:02:00.3: VF(n) BAR0 space: [mem 0xd0be0000-0xd0bfffff] (contains BAR0 for 8 VFs) [ 1.731259] pci 0000:02:00.3: reg 0x190: [mem 0xd0bc0000-0xd0bc3fff] [ 1.731263] pci 0000:02:00.3: VF(n) BAR3 space: [mem 0xd0bc0000-0xd0bdffff] (contains BAR3 for 8 VFs) [ 1.744088] pci 0000:00:01.1: PCI bridge to [bus 02-03] [ 1.750223] pci 0000:00:01.1: bridge window [io 0x3000-0x3fff] [ 1.750229] pci 0000:00:01.1: bridge window [mem 0xd0b00000-0xd0cfffff] [ 1.750398] acpiphp: Slot [2-2] registered [ 1.755483] pci 0000:04:00.0: [15b3:1003] type 00 class 0x028000 [ 1.755760] pci 0000:04:00.0: reg 0x10: [mem 0xd0e00000-0xd0efffff 64bit] [ 1.755963] pci 0000:04:00.0: reg 0x18: [mem 0xeb000000-0xeb7fffff 64bit pref] [ 1.757739] pci 0000:00:02.0: PCI bridge to [bus 04] [ 1.763582] pci 0000:00:02.0: bridge window [mem 0xd0e00000-0xd0efffff] [ 1.763590] pci 0000:00:02.0: bridge window [mem 0xeb000000-0xeb7fffff 64bit pref] [ 1.763748] acpiphp: Slot [2-3] registered [ 1.768659] pci 0000:00:02.2: PCI bridge to [bus 05] [ 1.774657] acpiphp: Slot [2-4] registered [ 1.779569] pci 0000:06:00.0: [1000:0087] type 00 class 0x010700 [ 1.779584] pci 0000:06:00.0: reg 0x10: [io 0x2000-0x20ff] [ 1.779599] pci 0000:06:00.0: reg 0x14: [mem 0xd0a40000-0xd0a4ffff 64bit] [ 1.779612] pci 0000:06:00.0: reg 0x1c: [mem 0xd0a00000-0xd0a3ffff 64bit] [ 1.779629] pci 0000:06:00.0: reg 0x30: [mem 0xd0900000-0xd09fffff pref] [ 1.779707] pci 0000:06:00.0: supports D1 D2 [ 1.782538] pci 0000:00:03.0: PCI bridge to [bus 06] [ 1.788381] pci 0000:00:03.0: bridge window [io 0x2000-0x2fff] [ 1.788387] pci 0000:00:03.0: bridge window [mem 0xd0900000-0xd0afffff] [ 1.788511] pci 0000:07:00.0: [8086:1d6b] type 00 class 0x010700 [ 1.788539] pci 0000:07:00.0: reg 0x10: [mem 0xebc00000-0xebc03fff 64bit pref] [ 1.788560] pci 0000:07:00.0: reg 0x18: [mem 0xeb800000-0xebbfffff 64bit pref] [ 1.788573] pci 0000:07:00.0: reg 0x20: [io 0x1000-0x10ff] [ 1.788710] pci 0000:07:00.0: reg 0x164: [mem 0xebc10000-0xebc13fff 64bit pref] [ 1.788715] pci 0000:07:00.0: VF(n) BAR0 space: [mem 0xebc10000-0xebc8bfff 64bit pref] (contains BAR0 for 31 VFs) [ 1.800924] pci 0000:07:00.3: [8086:1d70] type 00 class 0x0c0500 [ 1.800949] pci 0000:07:00.3: reg 0x10: [mem 0xd0d00000-0xd0d00fff] [ 1.800999] pci 0000:07:00.3: reg 0x20: [io 0x1100-0x111f] [ 1.801097] pci 0000:07:00.3: PME# supported from D0 D3hot D3cold [ 1.801200] pci 0000:00:11.0: PCI bridge to [bus 07] [ 1.807042] pci 0000:00:11.0: bridge window [io 0x1000-0x1fff] [ 1.807048] pci 0000:00:11.0: bridge window [mem 0xd0d00000-0xd0dfffff] [ 1.807058] pci 0000:00:11.0: bridge window [mem 0xeb800000-0xebcfffff 64bit pref] [ 1.807163] pci 0000:00:1c.0: PCI bridge to [bus 08] [ 1.813109] pci 0000:09:00.0: [102b:0522] type 00 class 0x030000 [ 1.813143] pci 0000:09:00.0: reg 0x10: [mem 0xea000000-0xeaffffff pref] [ 1.813163] pci 0000:09:00.0: reg 0x14: [mem 0xd0810000-0xd0813fff] [ 1.813184] pci 0000:09:00.0: reg 0x18: [mem 0xd0000000-0xd07fffff] [ 1.813257] pci 0000:09:00.0: reg 0x30: [mem 0xd0800000-0xd080ffff pref] [ 1.813404] pci 0000:09:00.0: System wakeup disabled by ACPI [ 1.822046] pci 0000:00:1c.7: PCI bridge to [bus 09] [ 1.827888] pci 0000:00:1c.7: bridge window [mem 0xd0000000-0xd08fffff] [ 1.827897] pci 0000:00:1c.7: bridge window [mem 0xea000000-0xeaffffff 64bit pref] [ 1.828010] pci 0000:00:1e.0: PCI bridge to [bus 0a] (subtractive decode) [ 1.835898] pci 0000:00:1e.0: bridge window [io 0x0000-0xbfff window] (subtractive decode) [ 1.835901] pci 0000:00:1e.0: bridge window [mem 0x000a0000-0x000bffff window] (subtractive decode) [ 1.835905] pci 0000:00:1e.0: bridge window [mem 0x000c0000-0x000c3fff window] (subtractive decode) [ 1.835908] pci 0000:00:1e.0: bridge window [mem 0x000c4000-0x000c7fff window] (subtractive decode) [ 1.835912] pci 0000:00:1e.0: bridge window [mem 0x000c8000-0x000cbfff window] (subtractive decode) [ 1.835915] pci 0000:00:1e.0: bridge window [mem 0x000cc000-0x000cffff window] (subtractive decode) [ 1.835918] pci 0000:00:1e.0: bridge window [mem 0x000d4000-0x000d7fff window] (subtractive decode) [ 1.835921] pci 0000:00:1e.0: bridge window [mem 0x000dc000-0x000dffff window] (subtractive decode) [ 1.835924] pci 0000:00:1e.0: bridge window [mem 0x000e0000-0x000e3fff window] (subtractive decode) [ 1.835928] pci 0000:00:1e.0: bridge window [mem 0x000e4000-0x000e7fff window] (subtractive decode) [ 1.835931] pci 0000:00:1e.0: bridge window [mem 0x000e8000-0x000ebfff window] (subtractive decode) [ 1.835954] pci 0000:00:1e.0: bridge window [mem 0x000ec000-0x000effff window] (subtractive decode) [ 1.835957] pci 0000:00:1e.0: bridge window [mem 0x000f0000-0x000fffff window] (subtractive decode) [ 1.835961] pci 0000:00:1e.0: bridge window [mem 0xd0000000-0xebffffff window] (subtractive decode) [ 1.835964] pci 0000:00:1e.0: bridge window [mem 0x380000000000-0x38007fffffff window] (subtractive decode) [ 1.836043] pci_bus 0000:00: on NUMA node 0 [ 1.837229] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 1.846250] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15) [ 1.855263] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 10 11 12 14 15) [ 1.864252] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 *11 12 14 15) [ 1.873246] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 *5 6 10 11 12 14 15) [ 1.882238] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15) [ 1.891215] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 *10 11 12 14 15) [ 1.900196] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 *10 11 12 14 15) [ 1.909363] ACPI: PCI Root Bridge [PCI1] (domain 0000 [bus 80-fe]) [ 1.916569] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 1.926464] acpi PNP0A08:01: _OSC: platform does not support [AER] [ 1.933899] acpi PNP0A08:01: _OSC: OS now controls [PCIeHotplug PME PCIeCapability] [ 1.942956] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 1.953004] PCI host bridge to bus 0000:80 [ 1.957871] pci_bus 0000:80: root bus resource [bus 80-fe] [ 1.964286] pci_bus 0000:80: root bus resource [io 0x03b0-0x03df window] [ 1.972163] pci_bus 0000:80: root bus resource [io 0xc000-0xffff window] [ 1.980043] pci_bus 0000:80: root bus resource [mem 0x000a0000-0x000bffff window] [ 1.988901] pci_bus 0000:80: root bus resource [mem 0xec000000-0xfbffffff window] [ 1.997759] pci_bus 0000:80: root bus resource [mem 0x380080000000-0x3800ffffffff window] [ 2.007416] pci 0000:80:01.0: [8086:3c02] type 01 class 0x060400 [ 2.007527] pci 0000:80:01.0: PME# supported from D0 D3hot D3cold [ 2.007601] pci 0000:80:01.0: System wakeup disabled by ACPI [ 2.014302] pci 0000:80:02.0: [8086:3c04] type 01 class 0x060400 [ 2.014410] pci 0000:80:02.0: PME# supported from D0 D3hot D3cold [ 2.014473] pci 0000:80:02.0: System wakeup disabled by ACPI [ 2.021161] pci 0000:80:03.0: [8086:3c08] type 01 class 0x060400 [ 2.021267] pci 0000:80:03.0: PME# supported from D0 D3hot D3cold [ 2.021335] pci 0000:80:03.0: System wakeup disabled by ACPI [ 2.028019] pci 0000:80:03.2: [8086:3c0a] type 01 class 0x060400 [ 2.028124] pci 0000:80:03.2: PME# supported from D0 D3hot D3cold [ 2.028186] pci 0000:80:03.2: System wakeup disabled by ACPI [ 2.034879] pci 0000:80:04.0: [8086:3c20] type 00 class 0x088000 [ 2.034903] pci 0000:80:04.0: reg 0x10: [mem 0xfbf70000-0xfbf73fff 64bit] [ 2.035078] pci 0000:80:04.1: [8086:3c21] type 00 class 0x088000 [ 2.035101] pci 0000:80:04.1: reg 0x10: [mem 0xfbf60000-0xfbf63fff 64bit] [ 2.035273] pci 0000:80:04.2: [8086:3c22] type 00 class 0x088000 [ 2.035295] pci 0000:80:04.2: reg 0x10: [mem 0xfbf50000-0xfbf53fff 64bit] [ 2.035463] pci 0000:80:04.3: [8086:3c23] type 00 class 0x088000 [ 2.035485] pci 0000:80:04.3: reg 0x10: [mem 0xfbf40000-0xfbf43fff 64bit] [ 2.035652] pci 0000:80:04.4: [8086:3c24] type 00 class 0x088000 [ 2.035675] pci 0000:80:04.4: reg 0x10: [mem 0xfbf30000-0xfbf33fff 64bit] [ 2.035848] pci 0000:80:04.5: [8086:3c25] type 00 class 0x088000 [ 2.035871] pci 0000:80:04.5: reg 0x10: [mem 0xfbf20000-0xfbf23fff 64bit] [ 2.036046] pci 0000:80:04.6: [8086:3c26] type 00 class 0x088000 [ 2.036069] pci 0000:80:04.6: reg 0x10: [mem 0xfbf10000-0xfbf13fff 64bit] [ 2.036235] pci 0000:80:04.7: [8086:3c27] type 00 class 0x088000 [ 2.036258] pci 0000:80:04.7: reg 0x10: [mem 0xfbf00000-0xfbf03fff 64bit] [ 2.036423] pci 0000:80:05.0: [8086:3c28] type 00 class 0x088000 [ 2.036579] pci 0000:80:05.2: [8086:3c2a] type 00 class 0x088000 [ 2.036734] pci 0000:80:05.4: [8086:3c2c] type 00 class 0x080020 [ 2.036752] pci 0000:80:05.4: reg 0x10: [mem 0xec200000-0xec200fff] [ 2.037003] pci 0000:80:01.0: PCI bridge to [bus 81] [ 2.042962] pci 0000:82:00.0: [1000:0087] type 00 class 0x010700 [ 2.042979] pci 0000:82:00.0: reg 0x10: [io 0xc000-0xc0ff] [ 2.042993] pci 0000:82:00.0: reg 0x14: [mem 0xec140000-0xec14ffff 64bit] [ 2.043008] pci 0000:82:00.0: reg 0x1c: [mem 0xec100000-0xec13ffff 64bit] [ 2.043026] pci 0000:82:00.0: reg 0x30: [mem 0xec000000-0xec0fffff pref] [ 2.043089] pci 0000:82:00.0: supports D1 D2 [ 2.044864] pci 0000:80:02.0: PCI bridge to [bus 82] [ 2.050703] pci 0000:80:02.0: bridge window [io 0xc000-0xcfff] [ 2.050709] pci 0000:80:02.0: bridge window [mem 0xec000000-0xec1fffff] [ 2.050790] pci 0000:80:03.0: PCI bridge to [bus 83] [ 2.056712] pci 0000:80:03.2: PCI bridge to [bus 84] [ 2.062595] pci_bus 0000:80: on NUMA node 1 [ 2.072507] ACPI: PCI Root Bridge [UCR0] (domain 0000 [bus 7f]) [ 2.079421] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.089108] acpi PNP0A03:00: _OSC: platform does not support [PCIeHotplug PME AER] [ 2.098109] acpi PNP0A03:00: _OSC: OS now controls [PCIeCapability] [ 2.105405] acpi PNP0A03:00: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.115150] PCI host bridge to bus 0000:7f [ 2.120009] pci_bus 0000:7f: root bus resource [bus 7f] [ 2.126152] pci 0000:7f:08.0: [8086:3c80] type 00 class 0x088000 [ 2.126277] pci 0000:7f:09.0: [8086:3c90] type 00 class 0x088000 [ 2.126369] pci 0000:7f:0a.0: [8086:3cc0] type 00 class 0x088000 [ 2.126451] pci 0000:7f:0a.1: [8086:3cc1] type 00 class 0x088000 [ 2.126529] pci 0000:7f:0a.2: [8086:3cc2] type 00 class 0x088000 [ 2.126609] pci 0000:7f:0a.3: [8086:3cd0] type 00 class 0x088000 [ 2.126693] pci 0000:7f:0b.0: [8086:3ce0] type 00 class 0x088000 [ 2.126773] pci 0000:7f:0b.3: [8086:3ce3] type 00 class 0x088000 [ 2.126862] pci 0000:7f:0c.0: [8086:3ce8] type 00 class 0x088000 [ 2.126942] pci 0000:7f:0c.1: [8086:3ce8] type 00 class 0x088000 [ 2.127018] pci 0000:7f:0c.2: [8086:3ce8] type 00 class 0x088000 [ 2.127095] pci 0000:7f:0c.3: [8086:3ce8] type 00 class 0x088000 [ 2.127183] pci 0000:7f:0c.6: [8086:3cf4] type 00 class 0x088000 [ 2.127268] pci 0000:7f:0c.7: [8086:3cf6] type 00 class 0x088000 [ 2.127352] pci 0000:7f:0d.0: [8086:3ce8] type 00 class 0x088000 [ 2.127432] pci 0000:7f:0d.1: [8086:3ce8] type 00 class 0x088000 [ 2.127511] pci 0000:7f:0d.2: [8086:3ce8] type 00 class 0x088000 [ 2.127589] pci 0000:7f:0d.3: [8086:3ce8] type 00 class 0x088000 [ 2.127670] pci 0000:7f:0d.6: [8086:3cf5] type 00 class 0x088000 [ 2.127748] pci 0000:7f:0e.0: [8086:3ca0] type 00 class 0x088000 [ 2.127831] pci 0000:7f:0e.1: [8086:3c46] type 00 class 0x110100 [ 2.127926] pci 0000:7f:0f.0: [8086:3ca8] type 00 class 0x088000 [ 2.128035] pci 0000:7f:0f.1: [8086:3c71] type 00 class 0x088000 [ 2.128136] pci 0000:7f:0f.2: [8086:3caa] type 00 class 0x088000 [ 2.128246] pci 0000:7f:0f.3: [8086:3cab] type 00 class 0x088000 [ 2.128348] pci 0000:7f:0f.4: [8086:3cac] type 00 class 0x088000 [ 2.128450] pci 0000:7f:0f.5: [8086:3cad] type 00 class 0x088000 [ 2.128553] pci 0000:7f:0f.6: [8086:3cae] type 00 class 0x088000 [ 2.128642] pci 0000:7f:10.0: [8086:3cb0] type 00 class 0x088000 [ 2.128744] pci 0000:7f:10.1: [8086:3cb1] type 00 class 0x088000 [ 2.128849] pci 0000:7f:10.2: [8086:3cb2] type 00 class 0x088000 [ 2.128950] pci 0000:7f:10.3: [8086:3cb3] type 00 class 0x088000 [ 2.129057] pci 0000:7f:10.4: [8086:3cb4] type 00 class 0x088000 [ 2.129164] pci 0000:7f:10.5: [8086:3cb5] type 00 class 0x088000 [ 2.129264] pci 0000:7f:10.6: [8086:3cb6] type 00 class 0x088000 [ 2.129364] pci 0000:7f:10.7: [8086:3cb7] type 00 class 0x088000 [ 2.129463] pci 0000:7f:11.0: [8086:3cb8] type 00 class 0x088000 [ 2.129557] pci 0000:7f:13.0: [8086:3ce4] type 00 class 0x088000 [ 2.129639] pci 0000:7f:13.1: [8086:3c43] type 00 class 0x110100 [ 2.129722] pci 0000:7f:13.4: [8086:3ce6] type 00 class 0x110100 [ 2.129802] pci 0000:7f:13.5: [8086:3c44] type 00 class 0x110100 [ 2.129882] pci 0000:7f:13.6: [8086:3c45] type 00 class 0x088000 [ 2.130065] ACPI: PCI Root Bridge [UCR1] (domain 0000 [bus ff]) [ 2.136977] acpi PNP0A03:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 2.146669] acpi PNP0A03:01: _OSC: platform does not support [PCIeHotplug PME AER] [ 2.160379] acpi PNP0A03:01: _OSC: OS now controls [PCIeCapability] [ 2.167674] acpi PNP0A03:01: FADT indicates ASPM is unsupported, using BIOS configuration [ 2.177422] PCI host bridge to bus 0000:ff [ 2.182287] pci_bus 0000:ff: root bus resource [bus ff] [ 2.188429] pci 0000:ff:08.0: [8086:3c80] type 00 class 0x088000 [ 2.188535] pci 0000:ff:09.0: [8086:3c90] type 00 class 0x088000 [ 2.188627] pci 0000:ff:0a.0: [8086:3cc0] type 00 class 0x088000 [ 2.188714] pci 0000:ff:0a.1: [8086:3cc1] type 00 class 0x088000 [ 2.188799] pci 0000:ff:0a.2: [8086:3cc2] type 00 class 0x088000 [ 2.188880] pci 0000:ff:0a.3: [8086:3cd0] type 00 class 0x088000 [ 2.188972] pci 0000:ff:0b.0: [8086:3ce0] type 00 class 0x088000 [ 2.189051] pci 0000:ff:0b.3: [8086:3ce3] type 00 class 0x088000 [ 2.189130] pci 0000:ff:0c.0: [8086:3ce8] type 00 class 0x088000 [ 2.189208] pci 0000:ff:0c.1: [8086:3ce8] type 00 class 0x088000 [ 2.189287] pci 0000:ff:0c.2: [8086:3ce8] type 00 class 0x088000 [ 2.189368] pci 0000:ff:0c.3: [8086:3ce8] type 00 class 0x088000 [ 2.189462] pci 0000:ff:0c.6: [8086:3cf4] type 00 class 0x088000 [ 2.189549] pci 0000:ff:0c.7: [8086:3cf6] type 00 class 0x088000 [ 2.189630] pci 0000:ff:0d.0: [8086:3ce8] type 00 class 0x088000 [ 2.189713] pci 0000:ff:0d.1: [8086:3ce8] type 00 class 0x088000 [ 2.189793] pci 0000:ff:0d.2: [8086:3ce8] type 00 class 0x088000 [ 2.189872] pci 0000:ff:0d.3: [8086:3ce8] type 00 class 0x088000 [ 2.189956] pci 0000:ff:0d.6: [8086:3cf5] type 00 class 0x088000 [ 2.190040] pci 0000:ff:0e.0: [8086:3ca0] type 00 class 0x088000 [ 2.190135] pci 0000:ff:0e.1: [8086:3c46] type 00 class 0x110100 [ 2.190235] pci 0000:ff:0f.0: [8086:3ca8] type 00 class 0x088000 [ 2.190342] pci 0000:ff:0f.1: [8086:3c71] type 00 class 0x088000 [ 2.190456] pci 0000:ff:0f.2: [8086:3caa] type 00 class 0x088000 [ 2.190563] pci 0000:ff:0f.3: [8086:3cab] type 00 class 0x088000 [ 2.190669] pci 0000:ff:0f.4: [8086:3cac] type 00 class 0x088000 [ 2.190770] pci 0000:ff:0f.5: [8086:3cad] type 00 class 0x088000 [ 2.190873] pci 0000:ff:0f.6: [8086:3cae] type 00 class 0x088000 [ 2.190968] pci 0000:ff:10.0: [8086:3cb0] type 00 class 0x088000 [ 2.191076] pci 0000:ff:10.1: [8086:3cb1] type 00 class 0x088000 [ 2.191185] pci 0000:ff:10.2: [8086:3cb2] type 00 class 0x088000 [ 2.191289] pci 0000:ff:10.3: [8086:3cb3] type 00 class 0x088000 [ 2.191393] pci 0000:ff:10.4: [8086:3cb4] type 00 class 0x088000 [ 2.191506] pci 0000:ff:10.5: [8086:3cb5] type 00 class 0x088000 [ 2.191611] pci 0000:ff:10.6: [8086:3cb6] type 00 class 0x088000 [ 2.191716] pci 0000:ff:10.7: [8086:3cb7] type 00 class 0x088000 [ 2.191821] pci 0000:ff:11.0: [8086:3cb8] type 00 class 0x088000 [ 2.191919] pci 0000:ff:13.0: [8086:3ce4] type 00 class 0x088000 [ 2.192000] pci 0000:ff:13.1: [8086:3c43] type 00 class 0x110100 [ 2.192085] pci 0000:ff:13.4: [8086:3ce6] type 00 class 0x110100 [ 2.192167] pci 0000:ff:13.5: [8086:3c44] type 00 class 0x110100 [ 2.192249] pci 0000:ff:13.6: [8086:3c45] type 00 class 0x088000 [ 2.192593] ACPI: Enabled 6 GPEs in block 00 to 3F [ 2.198534] vgaarb: device added: PCI:0000:09:00.0,decodes=io+mem,owns=io+mem,locks=none [ 2.208097] vgaarb: loaded [ 2.211403] vgaarb: bridge control possible 0000:09:00.0 [ 2.217845] SCSI subsystem initialized [ 2.222384] ACPI: bus type USB registered [ 2.227182] usbcore: registered new interface driver usbfs [ 2.233621] usbcore: registered new interface driver hub [ 2.240134] usbcore: registered new device driver usb [ 2.246628] PCI: Using ACPI for IRQ routing [ 2.258176] PCI: pci_cache_line_size set to 64 bytes [ 2.258529] e820: reserve RAM buffer [mem 0x0008d400-0x0008ffff] [ 2.258532] e820: reserve RAM buffer [mem 0xbb3c7000-0xbbffffff] [ 2.258535] e820: reserve RAM buffer [mem 0xbe000000-0xbfffffff] [ 2.258938] NetLabel: Initializing [ 2.263024] NetLabel: domain hash size = 128 [ 2.268177] NetLabel: protocols = UNLABELED CIPSOv4 [ 2.274040] NetLabel: unlabeled traffic allowed by default [ 2.280627] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0 [ 2.288604] hpet0: 8 comparators, 64-bit 14.318180 MHz counter [ 2.298471] amd_nb: Cannot enumerate AMD northbridges [ 2.304631] Switched to clocksource hpet [ 2.323959] pnp: PnP ACPI init [ 2.327702] ACPI: bus type PNP registered [ 2.332915] system 00:00: [io 0x0680-0x069f] has been reserved [ 2.339841] system 00:00: [io 0xffff] has been reserved [ 2.346076] system 00:00: [io 0xffff] has been reserved [ 2.352312] system 00:00: [io 0xffff] has been reserved [ 2.358549] system 00:00: [io 0x0400-0x0453] could not be reserved [ 2.365856] system 00:00: [io 0x0458-0x047f] has been reserved [ 2.372771] system 00:00: [io 0x0500-0x057f] has been reserved [ 2.379689] system 00:00: [io 0x0600-0x061f] has been reserved [ 2.386605] system 00:00: [io 0x0ca2-0x0ca5] has been reserved [ 2.393520] system 00:00: [io 0x0cf9] could not be reserved [ 2.400149] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.400211] pnp 00:01: Plug and Play ACPI device, IDs PNP0b00 (active) [ 2.400342] system 00:02: [io 0x0454-0x0457] has been reserved [ 2.407265] system 00:02: Plug and Play ACPI device, IDs INT3f0d PNP0c02 (active) [ 2.407657] pnp 00:03: [dma 0 disabled] [ 2.407760] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.408025] pnp 00:04: [dma 0 disabled] [ 2.408122] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 2.408759] system 00:05: [mem 0xfed1c000-0xfed1ffff] has been reserved [ 2.416463] system 00:05: [mem 0xebfff000-0xebffffff] has been reserved [ 2.424159] system 00:05: [mem 0xc0000000-0xcfffffff] has been reserved [ 2.431863] system 00:05: [mem 0xfed20000-0xfed3ffff] has been reserved [ 2.439558] system 00:05: [mem 0xfed45000-0xfed8ffff] has been reserved [ 2.447255] system 00:05: [mem 0xff000000-0xffffffff] could not be reserved [ 2.455343] system 00:05: [mem 0xfee00000-0xfeefffff] could not be reserved [ 2.463430] system 00:05: [mem 0xfec00000-0xfecfffff] could not be reserved [ 2.471515] system 00:05: [mem 0xd0f70000-0xd0f70fff] has been reserved [ 2.479206] system 00:05: Plug and Play ACPI device, IDs PNP0c02 (active) [ 2.479732] system 00:06: [mem 0x00000000-0x0009cfff] could not be reserved [ 2.487823] system 00:06: Plug and Play ACPI device, IDs PNP0c01 (active) [ 2.488355] pnp: PnP ACPI: found 7 devices [ 2.493233] ACPI: bus type PNP unregistered [ 2.506532] pci 0000:00:01.0: PCI bridge to [bus 01] [ 2.512399] pci 0000:00:01.1: PCI bridge to [bus 02-03] [ 2.518541] pci 0000:00:01.1: bridge window [io 0x3000-0x3fff] [ 2.525659] pci 0000:00:01.1: bridge window [mem 0xd0b00000-0xd0cfffff] [ 2.533558] pci 0000:00:02.0: PCI bridge to [bus 04] [ 2.539401] pci 0000:00:02.0: bridge window [mem 0xd0e00000-0xd0efffff] [ 2.547295] pci 0000:00:02.0: bridge window [mem 0xeb000000-0xeb7fffff 64bit pref] [ 2.556469] pci 0000:00:02.2: PCI bridge to [bus 05] [ 2.562330] pci 0000:00:03.0: PCI bridge to [bus 06] [ 2.568178] pci 0000:00:03.0: bridge window [io 0x2000-0x2fff] [ 2.575292] pci 0000:00:03.0: bridge window [mem 0xd0900000-0xd0afffff] [ 2.583191] pci 0000:00:11.0: PCI bridge to [bus 07] [ 2.589038] pci 0000:00:11.0: bridge window [io 0x1000-0x1fff] [ 2.596154] pci 0000:00:11.0: bridge window [mem 0xd0d00000-0xd0dfffff] [ 2.604048] pci 0000:00:11.0: bridge window [mem 0xeb800000-0xebcfffff 64bit pref] [ 2.613225] pci 0000:00:1c.0: PCI bridge to [bus 08] [ 2.619085] pci 0000:00:1c.7: PCI bridge to [bus 09] [ 2.624938] pci 0000:00:1c.7: bridge window [mem 0xd0000000-0xd08fffff] [ 2.632833] pci 0000:00:1c.7: bridge window [mem 0xea000000-0xeaffffff 64bit pref] [ 2.642003] pci 0000:00:1e.0: PCI bridge to [bus 0a] [ 2.647863] pci_bus 0000:00: resource 4 [io 0x0000-0xbfff window] [ 2.647867] pci_bus 0000:00: resource 5 [mem 0x000a0000-0x000bffff window] [ 2.647870] pci_bus 0000:00: resource 6 [mem 0x000c0000-0x000c3fff window] [ 2.647874] pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window] [ 2.647877] pci_bus 0000:00: resource 8 [mem 0x000c8000-0x000cbfff window] [ 2.647880] pci_bus 0000:00: resource 9 [mem 0x000cc000-0x000cffff window] [ 2.647883] pci_bus 0000:00: resource 10 [mem 0x000d4000-0x000d7fff window] [ 2.647887] pci_bus 0000:00: resource 11 [mem 0x000dc000-0x000dffff window] [ 2.647890] pci_bus 0000:00: resource 12 [mem 0x000e0000-0x000e3fff window] [ 2.647893] pci_bus 0000:00: resource 13 [mem 0x000e4000-0x000e7fff window] [ 2.647897] pci_bus 0000:00: resource 14 [mem 0x000e8000-0x000ebfff window] [ 2.647900] pci_bus 0000:00: resource 15 [mem 0x000ec000-0x000effff window] [ 2.647903] pci_bus 0000:00: resource 16 [mem 0x000f0000-0x000fffff window] [ 2.647907] pci_bus 0000:00: resource 17 [mem 0xd0000000-0xebffffff window] [ 2.647910] pci_bus 0000:00: resource 18 [mem 0x380000000000-0x38007fffffff window] [ 2.647914] pci_bus 0000:02: resource 0 [io 0x3000-0x3fff] [ 2.647917] pci_bus 0000:02: resource 1 [mem 0xd0b00000-0xd0cfffff] [ 2.647921] pci_bus 0000:04: resource 1 [mem 0xd0e00000-0xd0efffff] [ 2.647924] pci_bus 0000:04: resource 2 [mem 0xeb000000-0xeb7fffff 64bit pref] [ 2.647928] pci_bus 0000:06: resource 0 [io 0x2000-0x2fff] [ 2.647931] pci_bus 0000:06: resource 1 [mem 0xd0900000-0xd0afffff] [ 2.647935] pci_bus 0000:07: resource 0 [io 0x1000-0x1fff] [ 2.647938] pci_bus 0000:07: resource 1 [mem 0xd0d00000-0xd0dfffff] [ 2.647941] pci_bus 0000:07: resource 2 [mem 0xeb800000-0xebcfffff 64bit pref] [ 2.647945] pci_bus 0000:09: resource 1 [mem 0xd0000000-0xd08fffff] [ 2.647948] pci_bus 0000:09: resource 2 [mem 0xea000000-0xeaffffff 64bit pref] [ 2.647952] pci_bus 0000:0a: resource 4 [io 0x0000-0xbfff window] [ 2.647955] pci_bus 0000:0a: resource 5 [mem 0x000a0000-0x000bffff window] [ 2.647958] pci_bus 0000:0a: resource 6 [mem 0x000c0000-0x000c3fff window] [ 2.647961] pci_bus 0000:0a: resource 7 [mem 0x000c4000-0x000c7fff window] [ 2.647965] pci_bus 0000:0a: resource 8 [mem 0x000c8000-0x000cbfff window] [ 2.647968] pci_bus 0000:0a: resource 9 [mem 0x000cc000-0x000cffff window] [ 2.647971] pci_bus 0000:0a: resource 10 [mem 0x000d4000-0x000d7fff window] [ 2.647974] pci_bus 0000:0a: resource 11 [mem 0x000dc000-0x000dffff window] [ 2.647978] pci_bus 0000:0a: resource 12 [mem 0x000e0000-0x000e3fff window] [ 2.647981] pci_bus 0000:0a: resource 13 [mem 0x000e4000-0x000e7fff window] [ 2.647984] pci_bus 0000:0a: resource 14 [mem 0x000e8000-0x000ebfff window] [ 2.647988] pci_bus 0000:0a: resource 15 [mem 0x000ec000-0x000effff window] [ 2.647991] pci_bus 0000:0a: resource 16 [mem 0x000f0000-0x000fffff window] [ 2.647994] pci_bus 0000:0a: resource 17 [mem 0xd0000000-0xebffffff window] [ 2.647998] pci_bus 0000:0a: resource 18 [mem 0x380000000000-0x38007fffffff window] [ 2.648037] pci 0000:80:01.0: PCI bridge to [bus 81] [ 2.653893] pci 0000:80:02.0: PCI bridge to [bus 82] [ 2.659738] pci 0000:80:02.0: bridge window [io 0xc000-0xcfff] [ 2.666850] pci 0000:80:02.0: bridge window [mem 0xec000000-0xec1fffff] [ 2.674748] pci 0000:80:03.0: PCI bridge to [bus 83] [ 2.680603] pci 0000:80:03.2: PCI bridge to [bus 84] [ 2.686454] pci_bus 0000:80: resource 4 [io 0x03b0-0x03df window] [ 2.686458] pci_bus 0000:80: resource 5 [io 0xc000-0xffff window] [ 2.686461] pci_bus 0000:80: resource 6 [mem 0x000a0000-0x000bffff window] [ 2.686464] pci_bus 0000:80: resource 7 [mem 0xec000000-0xfbffffff window] [ 2.686468] pci_bus 0000:80: resource 8 [mem 0x380080000000-0x3800ffffffff window] [ 2.686472] pci_bus 0000:82: resource 0 [io 0xc000-0xcfff] [ 2.686475] pci_bus 0000:82: resource 1 [mem 0xec000000-0xec1fffff] [ 2.686835] NET: Registered protocol family 2 [ 2.694370] TCP established hash table entries: 262144 (order: 9, 2097152 bytes) [ 2.704098] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 2.712163] TCP: Hash tables configured (established 262144 bind 65536) [ 2.719997] TCP: reno registered [ 2.724061] UDP hash table entries: 16384 (order: 7, 524288 bytes) [ 2.731520] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes) [ 2.740518] NET: Registered protocol family 1 [ 3.845633] pci 0000:00:1a.0: EHCI: BIOS handoff failed (BIOS bug?) 01010001 [ 4.953668] pci 0000:00:1d.0: EHCI: BIOS handoff failed (BIOS bug?) 01010001 [ 4.962145] PCI: CLS mismatch (64 != 32), using 64 bytes [ 4.962171] pci 0000:09:00.0: Boot video device [ 4.962455] Unpacking initramfs... [ 5.580956] Freeing initrd memory: 19724k freed [ 5.594640] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 5.602172] software IO TLB [mem 0xb73c7000-0xbb3c7000] (64MB) mapped at [ffff8800b73c7000-ffff8800bb3c6fff] [ 5.614151] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 163840 ms ovfl timer [ 5.623541] RAPL PMU: hw unit of domain pp0-core 2^-16 Joules [ 5.630273] RAPL PMU: hw unit of domain package 2^-16 Joules [ 5.636903] RAPL PMU: hw unit of domain dram 2^-16 Joules [ 5.654969] sha1_ssse3: Using AVX optimized SHA-1 implementation [ 5.662101] sha256_ssse3: Using AVX optimized SHA-256 implementation [ 5.674573] futex hash table entries: 65536 (order: 10, 4194304 bytes) [ 5.683151] Initialise system trusted keyring [ 5.688472] audit: initializing netlink socket (disabled) [ 5.694863] type=2000 audit(1513210121.295:1): initialized [ 5.751618] HugeTLB registered 1 GB page size, pre-allocated 0 pages [ 5.759026] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 5.768559] zpool: loaded [ 5.771800] zbud: loaded [ 5.775885] VFS: Disk quotas dquot_6.6.0 [ 5.781077] Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 5.789993] msgmni has been set to 32768 [ 5.795084] Key type big_key registered [ 5.799700] SELinux: Registering netfilter hooks [ 5.802229] NET: Registered protocol family 38 [ 5.807529] Key type asymmetric registered [ 5.812430] Asymmetric key parser 'x509' registered [ 5.818345] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250) [ 5.827428] io scheduler noop registered [ 5.832128] io scheduler deadline registered (default) [ 5.838438] io scheduler cfq registered [ 5.843794] pcieport 0000:00:01.0: irq 25 for MSI/MSI-X [ 5.844092] pcieport 0000:00:01.1: irq 26 for MSI/MSI-X [ 5.844375] pcieport 0000:00:02.0: irq 27 for MSI/MSI-X [ 5.844602] pcieport 0000:00:02.2: irq 28 for MSI/MSI-X [ 5.844899] pcieport 0000:00:03.0: irq 29 for MSI/MSI-X [ 5.845185] pcieport 0000:00:11.0: irq 30 for MSI/MSI-X [ 5.845427] pcieport 0000:00:1c.0: irq 31 for MSI/MSI-X [ 5.845755] pcieport 0000:00:1c.7: irq 32 for MSI/MSI-X [ 5.846274] pcieport 0000:80:01.0: irq 34 for MSI/MSI-X [ 5.846584] pcieport 0000:80:02.0: irq 35 for MSI/MSI-X [ 5.846880] pcieport 0000:80:03.0: irq 36 for MSI/MSI-X [ 5.847158] pcieport 0000:80:03.2: irq 37 for MSI/MSI-X [ 5.847335] pcieport 0000:00:01.0: Signaling PME through PCIe PME interrupt [ 5.855450] pcie_pme 0000:00:01.0:pcie01: service driver pcie_pme loaded [ 5.855481] pcieport 0000:00:01.1: Signaling PME through PCIe PME interrupt [ 5.863580] pci 0000:02:00.0: Signaling PME through PCIe PME interrupt [ 5.871196] pci 0000:02:00.1: Signaling PME through PCIe PME interrupt [ 5.878813] pci 0000:02:00.2: Signaling PME through PCIe PME interrupt [ 5.886416] pci 0000:02:00.3: Signaling PME through PCIe PME interrupt [ 5.894024] pcie_pme 0000:00:01.1:pcie01: service driver pcie_pme loaded [ 5.894054] pcieport 0000:00:02.0: Signaling PME through PCIe PME interrupt [ 5.902152] pci 0000:04:00.0: Signaling PME through PCIe PME interrupt [ 5.909775] pcie_pme 0000:00:02.0:pcie01: service driver pcie_pme loaded [ 5.909812] pcieport 0000:00:02.2: Signaling PME through PCIe PME interrupt [ 5.917920] pcie_pme 0000:00:02.2:pcie01: service driver pcie_pme loaded [ 5.917957] pcieport 0000:00:03.0: Signaling PME through PCIe PME interrupt [ 5.926055] pci 0000:06:00.0: Signaling PME through PCIe PME interrupt [ 5.933661] pcie_pme 0000:00:03.0:pcie01: service driver pcie_pme loaded [ 5.933705] pcieport 0000:00:11.0: Signaling PME through PCIe PME interrupt [ 5.941805] pci 0000:07:00.0: Signaling PME through PCIe PME interrupt [ 5.949406] pci 0000:07:00.3: Signaling PME through PCIe PME interrupt [ 5.957021] pcie_pme 0000:00:11.0:pcie01: service driver pcie_pme loaded [ 5.957069] pcieport 0000:00:1c.0: Signaling PME through PCIe PME interrupt [ 5.965174] pcie_pme 0000:00:1c.0:pcie01: service driver pcie_pme loaded [ 5.965218] pcieport 0000:00:1c.7: Signaling PME through PCIe PME interrupt [ 5.973326] pci 0000:09:00.0: Signaling PME through PCIe PME interrupt [ 5.980932] pcie_pme 0000:00:1c.7:pcie01: service driver pcie_pme loaded [ 5.980962] pcieport 0000:80:01.0: Signaling PME through PCIe PME interrupt [ 5.989062] pcie_pme 0000:80:01.0:pcie01: service driver pcie_pme loaded [ 5.989110] pcieport 0000:80:02.0: Signaling PME through PCIe PME interrupt [ 5.997202] pci 0000:82:00.0: Signaling PME through PCIe PME interrupt [ 6.004832] pcie_pme 0000:80:02.0:pcie01: service driver pcie_pme loaded [ 6.004866] pcieport 0000:80:03.0: Signaling PME through PCIe PME interrupt [ 6.012964] pcie_pme 0000:80:03.0:pcie01: service driver pcie_pme loaded [ 6.013011] pcieport 0000:80:03.2: Signaling PME through PCIe PME interrupt [ 6.021113] pcie_pme 0000:80:03.2:pcie01: service driver pcie_pme loaded [ 6.021234] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 6.027798] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 6.035673] intel_idle: MWAIT substates: 0x21120 [ 6.035676] intel_idle: v0.4.1 model 0x2D [ 6.035964] intel_idle: lapic_timer_reliable_states 0xffffffff [ 6.036090] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.044885] ACPI: Power Button [PWRF] [ 6.049328] ACPI: Requesting acpi_cpufreq [ 6.090242] ERST: Error Record Serialization Table (ERST) support is initialized. [ 6.099152] pstore: Registered erst as persistent store backend [ 6.106242] GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC. [ 6.115103] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.143140] 00:03: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A [ 6.170398] 00:04: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A [ 6.177370] Non-volatile memory driver v1.3 [ 6.182400] Linux agpgart interface v0.103 [ 6.187452] crash memory driver: version 1.1 [ 6.192706] rdac: device handler registered [ 6.197736] hp_sw: device handler registered [ 6.202817] emc: device handler registered [ 6.207698] alua: device handler registered [ 6.212720] libphy: Fixed MDIO Bus: probed [ 6.217643] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 6.225254] ehci-pci: EHCI PCI platform driver [ 6.230774] ehci-pci 0000:00:1a.0: EHCI Host Controller [ 6.236945] ehci-pci 0000:00:1a.0: new USB bus registered, assigned bus number 1 [ 6.245721] ehci-pci 0000:00:1a.0: debug port 2 [ 6.254997] ehci-pci 0000:00:1a.0: cache line size of 64 is not supported [ 6.255012] ehci-pci 0000:00:1a.0: irq 22, io mem 0xd0f20000 [ 6.266706] ehci-pci 0000:00:1a.0: USB 2.0 started, EHCI 1.00 [ 6.273466] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 6.281353] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 6.289922] usb usb1: Product: EHCI Host Controller [ 6.295669] usb usb1: Manufacturer: Linux 3.10.0-693.5.2.el7_lustre.x86_64 ehci_hcd [ 6.304717] usb usb1: SerialNumber: 0000:00:1a.0 [ 6.314834] hub 1-0:1.0: USB hub found [ 6.319307] hub 1-0:1.0: 2 ports detected [ 6.324333] ehci-pci 0000:00:1d.0: EHCI Host Controller [ 6.330506] ehci-pci 0000:00:1d.0: new USB bus registered, assigned bus number 2 [ 6.339279] ehci-pci 0000:00:1d.0: debug port 2 [ 6.348525] ehci-pci 0000:00:1d.0: cache line size of 64 is not supported [ 6.348545] ehci-pci 0000:00:1d.0: irq 20, io mem 0xd0f10000 [ 6.360709] ehci-pci 0000:00:1d.0: USB 2.0 started, EHCI 1.00 [ 6.367445] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 [ 6.375317] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 6.383887] usb usb2: Product: EHCI Host Controller [ 6.389625] usb usb2: Manufacturer: Linux 3.10.0-693.5.2.el7_lustre.x86_64 ehci_hcd [ 6.398672] usb usb2: SerialNumber: 0000:00:1d.0 [ 6.404185] hub 2-0:1.0: USB hub found [ 6.408657] hub 2-0:1.0: 2 ports detected [ 6.413528] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver [ 6.420753] ohci-pci: OHCI PCI platform driver [ 6.426035] uhci_hcd: USB Universal Host Controller Interface driver [ 6.433510] usbcore: registered new interface driver usbserial [ 6.440342] usbcore: registered new interface driver usbserial_generic [ 6.447955] usbserial: USB Serial support registered for generic [ 6.454990] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 6.625726] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 6.653746] tsc: Refined TSC clocksource calibration: 2693.509 MHz [ 7.501946] i8042: No controller found [ 7.506489] Switched to clocksource tsc [ 7.506502] mousedev: PS/2 mouse device common for all mice [ 7.506627] rtc_cmos 00:01: RTC can wake from S4 [ 7.506893] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 [ 7.506950] rtc_cmos 00:01: alarms up to one month, y3k, 242 bytes nvram, hpet irqs [ 7.507008] Intel P-state driver initializing. [ 7.512836] cpuidle: using governor menu [ 7.513792] hidraw: raw HID events driver (C) Jiri Kosina [ 7.513951] usbcore: registered new interface driver usbhid [ 7.513952] usbhid: USB HID core driver [ 7.514073] drop_monitor: Initializing network drop monitor service [ 7.514754] TCP: cubic registered [ 7.514765] Initializing XFRM netlink socket [ 7.515410] NET: Registered protocol family 10 [ 7.517084] NET: Registered protocol family 17 [ 7.518410] microcode: CPU0 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518422] microcode: CPU1 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518434] microcode: CPU2 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518445] microcode: CPU3 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518457] microcode: CPU4 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518468] microcode: CPU5 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518480] microcode: CPU6 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518492] microcode: CPU7 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518507] microcode: CPU8 sig=0x206d7, pf=0x1, revision=0x710 [ 7.518518] microcode: CPU9 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523055] microcode: CPU10 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523065] microcode: CPU11 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523073] microcode: CPU12 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523084] microcode: CPU13 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523108] microcode: CPU14 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523117] microcode: CPU15 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523128] microcode: CPU16 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523138] microcode: CPU17 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523148] microcode: CPU18 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523157] microcode: CPU19 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523171] microcode: CPU20 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523181] microcode: CPU21 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523191] microcode: CPU22 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523199] microcode: CPU23 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523207] microcode: CPU24 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523215] microcode: CPU25 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523225] microcode: CPU26 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523233] microcode: CPU27 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523242] microcode: CPU28 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523250] microcode: CPU29 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523258] microcode: CPU30 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523265] microcode: CPU31 sig=0x206d7, pf=0x1, revision=0x710 [ 7.523324] microcode: Microcode Update Driver: v2.01 , Peter Oruba [ 7.523494] PM: Hibernation image not present or could not be loaded. [ 7.523497] Loading compiled-in X.509 certificates [ 7.523555] Loaded X.509 cert 'CentOS Linux kpatch signing key: ea0413152cde1d98ebdca3fe6f0230904c9ef717' [ 7.523575] Loaded X.509 cert 'CentOS Linux Driver update signing key: 7f421ee0ab69461574bb358861dbe77762a4201b' [ 7.524572] Loaded X.509 cert 'CentOS Linux kernel signing key: 7ea479bc11aef19a3d98df1006d285ad571191bf' [ 7.524601] registered taskstats version 1 [ 7.527848] Key type trusted registered [ 7.530759] Key type encrypted registered [ 7.533939] IMA: No TPM chip found, activating TPM-bypass! [ 7.535339] Magic number: 13:150:102 [ 7.535632] memory memory251: hash matches [ 7.539264] rtc_cmos 00:01: setting system clock to 2017-12-14 00:08:47 UTC (1513210127) [ 7.556755] usb 2-1: new high-speed USB device number 2 using ehci-pci [ 7.571112] usb 1-1: New USB device found, idVendor=8087, idProduct=0024 [ 7.571113] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 7.571355] hub 1-1:1.0: USB hub found [ 7.571458] hub 1-1:1.0: 6 ports detected [ 7.672109] usb 2-1: New USB device found, idVendor=8087, idProduct=0024 [ 7.672110] usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 [ 7.672361] hub 2-1:1.0: USB hub found [ 7.672468] hub 2-1:1.0: 8 ports detected [ 7.947764] usb 2-1.4: new full-speed USB device number 3 using ehci-pci [ 7.973750] Freeing unused kernel memory: 1764k freed [ 7.985008] random: systemd: uninitialized urandom read (16 bytes read) [ 7.995076] random: systemd: uninitialized urandom read (16 bytes read) [ 8.002809] random: systemd: uninitialized urandom read (16 bytes read) [ 8.012654] systemd[1]: systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN) [ 8.034671] systemd[1]: Detected architecture x86-64. [ 8.040614] systemd[1]: Running in initial RAM disk. [ 8.049946] usb 2-1.4: New USB device found, idVendor=046b, idProduct=ff10 [ 8.059447] usb 2-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [ 8.069630] usb 2-1.4: Product: Virtual Keyboard and Mouse [ 8.077558] usb 2-1.4: Manufacturer: American Megatrends Inc. [ 8.084366] usb 2-1.4: SerialNumber: serial [ 8.084371] systemd[1]: Set hostname to . [ 8.098549] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.0/input/input1 [ 8.141903] random: systemd: uninitialized urandom read (16 bytes read) [ 8.149670] random: systemd: uninitialized urandom read (16 bytes read) [ 8.157428] random: systemd: uninitialized urandom read (16 bytes read) [ 8.164161] hid-generic 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input0 [ 8.165507] input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.4/2-1.4:1.1/input/input2 [ 8.165686] hid-generic 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1d.0-1.4/input1 [ 8.214847] random: systemd: uninitialized urandom read (16 bytes read) [ 8.223415] random: systemd: uninitialized urandom read (16 bytes read) [ 8.231320] random: systemd: uninitialized urandom read (16 bytes read) [ 8.239257] random: systemd: uninitialized urandom read (16 bytes read) [ 8.256934] systemd[1]: Created slice Root Slice. [ 8.262549] systemd[1]: Starting Root Slice. [ 8.274886] systemd[1]: Listening on udev Control Socket. [ 8.281218] systemd[1]: Starting udev Control Socket. [ 8.291906] systemd[1]: Reached target Timers. [ 8.297197] systemd[1]: Starting Timers. [ 8.308936] systemd[1]: Listening on Journal Socket. [ 8.314816] systemd[1]: Starting Journal Socket. [ 8.326921] systemd[1]: Listening on udev Kernel Socket. [ 8.333180] systemd[1]: Starting udev Kernel Socket. [ 8.343893] systemd[1]: Reached target Sockets. [ 8.349269] systemd[1]: Starting Sockets. [ 8.358901] systemd[1]: Reached target Swap. [ 8.363983] systemd[1]: Starting Swap. [ 8.374894] systemd[1]: Reached target Local File Systems. [ 8.381343] systemd[1]: Starting Local File Systems. [ 8.393926] systemd[1]: Created slice System Slice. [ 8.399704] systemd[1]: Starting System Slice. [ 8.405834] systemd[1]: Starting Create list of required static device nodes for the current kernel... [ 8.427699] systemd[1]: Starting Journal Service... [ 8.439554] systemd[1]: Starting Setup Virtual Console... [ 8.451709] systemd[1]: Starting dracut cmdline hook... [ 8.473678] systemd[1]: Starting Load Kernel Modules... [ 8.489885] systemd[1]: Reached target Slices. [ 8.495177] systemd[1]: Starting Slices. [ 8.504847] systemd[1]: Started Journal Service. [ 8.700076] mlx_compat: loading out-of-tree module taints kernel. [ 8.708185] mlx_compat: module verification failed: signature and/or required key missing - tainting kernel [ 8.722284] Compat-mlnx-ofed backport release: f36c870 [ 8.728908] Backport based on mlnx_ofed/mlnx-ofa_kernel-4.0.git f36c870 [ 8.737172] compat.git: mlnx_ofed/mlnx-ofa_kernel-4.0.git [ 8.746157] dca service started, version 1.12.1 [ 8.754704] libata version 3.00 loaded. [ 8.755212] pps_core: LinuxPPS API ver. 1 registered [ 8.761054] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 8.772752] [drm] Initialized [ 8.772847] mlx4_core: Mellanox ConnectX core driver v4.2-1.0.0 [ 8.772879] mlx4_core: Initializing 0000:04:00.0 [ 8.774504] mpt2sas version 20.103.00.00 loaded [ 8.774595] mpt2sas 0000:06:00.0: can't disable ASPM; OS doesn't have ASPM control [ 8.774766] mpt2sas_cm0: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32511224 kB) [ 8.775063] PTP clock support registered [ 8.775980] ahci 0000:00:1f.2: version 3.0 [ 8.776336] ahci 0000:00:1f.2: irq 40 for MSI/MSI-X [ 8.776436] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3f impl SATA mode [ 8.776438] ahci 0000:00:1f.2: flags: 64bit ncq sntf pm led clo pio slum part ems apst [ 8.786302] scsi host1: ahci [ 8.786440] scsi host2: ahci [ 8.786516] scsi host3: ahci [ 8.786587] scsi host4: ahci [ 8.786726] scsi host5: ahci [ 8.786814] scsi host6: ahci [ 8.786848] ata1: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00100 irq 40 [ 8.786849] ata2: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00180 irq 40 [ 8.786852] ata3: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00200 irq 40 [ 8.786854] ata4: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00280 irq 40 [ 8.786861] ata5: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00300 irq 40 [ 8.786868] ata6: SATA max UDMA/133 abar m2048@0xd0f00000 port 0xd0f00380 irq 40 [ 8.829320] mpt2sas_cm0: MSI-X vectors supported: 16, no of cores: 32, max_msix_vectors: -1 [ 8.829402] mpt2sas 0000:06:00.0: irq 41 for MSI/MSI-X [ 8.829468] mpt2sas 0000:06:00.0: irq 42 for MSI/MSI-X [ 8.829530] mpt2sas 0000:06:00.0: irq 43 for MSI/MSI-X [ 8.829590] mpt2sas 0000:06:00.0: irq 44 for MSI/MSI-X [ 8.829651] mpt2sas 0000:06:00.0: irq 45 for MSI/MSI-X [ 8.829710] mpt2sas 0000:06:00.0: irq 46 for MSI/MSI-X [ 8.829771] mpt2sas 0000:06:00.0: irq 47 for MSI/MSI-X [ 8.829840] mpt2sas 0000:06:00.0: irq 48 for MSI/MSI-X [ 8.829901] mpt2sas 0000:06:00.0: irq 49 for MSI/MSI-X [ 8.829964] mpt2sas 0000:06:00.0: irq 50 for MSI/MSI-X [ 8.830024] mpt2sas 0000:06:00.0: irq 51 for MSI/MSI-X [ 8.830084] mpt2sas 0000:06:00.0: irq 52 for MSI/MSI-X [ 8.830145] mpt2sas 0000:06:00.0: irq 53 for MSI/MSI-X [ 8.830208] mpt2sas 0000:06:00.0: irq 54 for MSI/MSI-X [ 8.830268] mpt2sas 0000:06:00.0: irq 55 for MSI/MSI-X [ 8.830330] mpt2sas 0000:06:00.0: irq 56 for MSI/MSI-X [ 8.830438] mpt2sas0-msix0: PCI-MSI-X enabled: IRQ 41 [ 8.830439] mpt2sas0-msix1: PCI-MSI-X enabled: IRQ 42 [ 8.830440] mpt2sas0-msix2: PCI-MSI-X enabled: IRQ 43 [ 8.830440] mpt2sas0-msix3: PCI-MSI-X enabled: IRQ 44 [ 8.830441] mpt2sas0-msix4: PCI-MSI-X enabled: IRQ 45 [ 8.830441] mpt2sas0-msix5: PCI-MSI-X enabled: IRQ 46 [ 8.830442] mpt2sas0-msix6: PCI-MSI-X enabled: IRQ 47 [ 8.830442] mpt2sas0-msix7: PCI-MSI-X enabled: IRQ 48 [ 8.830442] mpt2sas0-msix8: PCI-MSI-X enabled: IRQ 49 [ 8.830443] mpt2sas0-msix9: PCI-MSI-X enabled: IRQ 50 [ 8.830444] mpt2sas0-msix10: PCI-MSI-X enabled: IRQ 51 [ 8.830444] mpt2sas0-msix11: PCI-MSI-X enabled: IRQ 52 [ 8.830444] mpt2sas0-msix12: PCI-MSI-X enabled: IRQ 53 [ 8.830445] mpt2sas0-msix13: PCI-MSI-X enabled: IRQ 54 [ 8.830445] mpt2sas0-msix14: PCI-MSI-X enabled: IRQ 55 [ 8.830446] mpt2sas0-msix15: PCI-MSI-X enabled: IRQ 56 [ 8.830447] mpt2sas_cm0: iomem(0x00000000d0a40000), mapped(0xffffc90003b00000), size(65536) [ 8.830448] mpt2sas_cm0: ioport(0x0000000000002000), size(256) [ 8.878287] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k [ 8.878288] igb: Copyright (c) 2007-2014 Intel Corporation. [ 8.878290] isci: Intel(R) C600 SAS Controller Driver - version 1.2.0 [ 8.878338] isci 0000:07:00.0: driver configured for rev: 6 silicon [ 8.878344] isci 0000:07:00.0: OEM parameter table found in OROM [ 8.878346] isci 0000:07:00.0: OEM SAS parameters (version: 1.1) loaded (platform) [ 8.878946] igb 0000:02:00.0: irq 58 for MSI/MSI-X [ 8.879034] igb 0000:02:00.0: irq 58 for MSI/MSI-X [ 8.879096] igb 0000:02:00.0: irq 59 for MSI/MSI-X [ 8.879157] igb 0000:02:00.0: irq 60 for MSI/MSI-X [ 8.879218] igb 0000:02:00.0: irq 61 for MSI/MSI-X [ 8.879279] igb 0000:02:00.0: irq 62 for MSI/MSI-X [ 8.879339] igb 0000:02:00.0: irq 63 for MSI/MSI-X [ 8.879400] igb 0000:02:00.0: irq 64 for MSI/MSI-X [ 8.879469] igb 0000:02:00.0: irq 65 for MSI/MSI-X [ 8.879529] igb 0000:02:00.0: irq 66 for MSI/MSI-X [ 8.879556] igb 0000:02:00.0: PHY reset is blocked due to SOL/IDER session. [ 8.879568] isci 0000:07:00.0: SCU controller 0: phy 3-0 cables: {short, short, short, short} [ 8.882048] scsi host7: isci [ 8.882600] isci 0000:07:00.0: irq 67 for MSI/MSI-X [ 8.882665] isci 0000:07:00.0: irq 68 for MSI/MSI-X [ 8.887800] mpt2sas_cm0: sending message unit reset !! [ 8.889309] mpt2sas_cm0: message unit reset: SUCCESS [ 8.915344] igb 0000:02:00.0: added PHC on eth0 [ 8.915346] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection [ 8.915347] igb 0000:02:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:23:29 [ 8.915420] igb 0000:02:00.0: eth0: PBA No: 100000-000 [ 8.915421] igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 8.915871] igb 0000:02:00.1: irq 70 for MSI/MSI-X [ 8.915957] igb 0000:02:00.1: irq 70 for MSI/MSI-X [ 8.916019] igb 0000:02:00.1: irq 71 for MSI/MSI-X [ 8.916082] igb 0000:02:00.1: irq 72 for MSI/MSI-X [ 8.916143] igb 0000:02:00.1: irq 73 for MSI/MSI-X [ 8.916206] igb 0000:02:00.1: irq 74 for MSI/MSI-X [ 8.916265] igb 0000:02:00.1: irq 75 for MSI/MSI-X [ 8.916324] igb 0000:02:00.1: irq 76 for MSI/MSI-X [ 8.916389] igb 0000:02:00.1: irq 77 for MSI/MSI-X [ 8.916450] igb 0000:02:00.1: irq 78 for MSI/MSI-X [ 8.916477] igb 0000:02:00.1: PHY reset is blocked due to SOL/IDER session. [ 8.963036] igb 0000:02:00.1: added PHC on eth1 [ 8.963037] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection [ 8.963039] igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:23:2a [ 8.963112] igb 0000:02:00.1: eth1: PBA No: 100000-000 [ 8.963113] igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 8.968675] mpt2sas_cm0: Allocated physical memory: size(18690 kB) [ 8.968676] mpt2sas_cm0: Current Controller Queue Depth(10104),Max Controller Queue Depth(10240) [ 8.968676] mpt2sas_cm0: Scatter Gather Elements per IO(128) [ 8.985020] igb 0000:02:00.2: irq 80 for MSI/MSI-X [ 8.985110] igb 0000:02:00.2: irq 80 for MSI/MSI-X [ 8.985171] igb 0000:02:00.2: irq 81 for MSI/MSI-X [ 8.985232] igb 0000:02:00.2: irq 82 for MSI/MSI-X [ 8.985294] igb 0000:02:00.2: irq 83 for MSI/MSI-X [ 8.985356] igb 0000:02:00.2: irq 84 for MSI/MSI-X [ 8.985418] igb 0000:02:00.2: irq 85 for MSI/MSI-X [ 8.985484] igb 0000:02:00.2: irq 86 for MSI/MSI-X [ 8.985544] igb 0000:02:00.2: irq 87 for MSI/MSI-X [ 8.985605] igb 0000:02:00.2: irq 88 for MSI/MSI-X [ 9.020850] mpt2sas_cm0: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05), BiosVersion(07.39.00.00) [ 9.020851] mpt2sas_cm0: Protocol=( [ 9.020851] Initiator [ 9.020852] ,Target [ 9.020852] ), [ 9.020852] Capabilities=( [ 9.020853] TLR [ 9.020853] ,EEDP [ 9.020854] ,Snapshot Buffer [ 9.020854] ,Diag Trace Buffer [ 9.020854] ,Task Set Full [ 9.020855] ,NCQ [ 9.020855] ) [ 9.020900] scsi host0: Fusion MPT SAS Host [ 9.021283] mpt2sas 0000:82:00.0: can't disable ASPM; OS doesn't have ASPM control [ 9.021386] mpt2sas_cm0: sending port enable !! [ 9.021650] random: fast init done [ 9.021658] mpt2sas_cm1: 64 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (32511224 kB) [ 9.023986] mpt2sas_cm0: host_add: handle(0x0001), sas_addr(0x500605b005bb2be0), phys(8) [ 9.026290] mpt2sas_cm0: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x500304800050553f), phys(38) [ 9.038314] igb 0000:02:00.2: added PHC on eth2 [ 9.038315] igb 0000:02:00.2: Intel(R) Gigabit Ethernet Network Connection [ 9.038316] igb 0000:02:00.2: eth2: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:23:2b [ 9.038389] igb 0000:02:00.2: eth2: PBA No: 100000-000 [ 9.038390] igb 0000:02:00.2: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 9.041841] mpt2sas_cm0: expander_add: handle(0x0020), parent(0x0002), sas_addr(0x500304800040543f), phys(38) [ 9.053017] igb 0000:02:00.3: irq 91 for MSI/MSI-X [ 9.053106] igb 0000:02:00.3: irq 91 for MSI/MSI-X [ 9.053174] igb 0000:02:00.3: irq 92 for MSI/MSI-X [ 9.053239] igb 0000:02:00.3: irq 93 for MSI/MSI-X [ 9.053311] igb 0000:02:00.3: irq 94 for MSI/MSI-X [ 9.053376] igb 0000:02:00.3: irq 95 for MSI/MSI-X [ 9.053441] igb 0000:02:00.3: irq 96 for MSI/MSI-X [ 9.053506] igb 0000:02:00.3: irq 97 for MSI/MSI-X [ 9.053570] igb 0000:02:00.3: irq 98 for MSI/MSI-X [ 9.053636] igb 0000:02:00.3: irq 99 for MSI/MSI-X [ 9.064807] mpt2sas_cm0: port enable: SUCCESS [ 9.065899] scsi 0:0:0:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.065904] scsi 0:0:0:0: SSP: handle(0x000a), sas_addr(0x5000c5008429daa5), phy(8), device_name(0x5000c5008429daa4) [ 9.065905] scsi 0:0:0:0: SSP: enclosure_logical_id(0x500304800050553f), slot(0) [ 9.065908] scsi 0:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.075819] mpt2sas_cm1: MSI-X vectors supported: 16, no of cores: 32, max_msix_vectors: -1 [ 9.075880] mpt2sas 0000:82:00.0: irq 100 for MSI/MSI-X [ 9.075904] mpt2sas 0000:82:00.0: irq 101 for MSI/MSI-X [ 9.075926] mpt2sas 0000:82:00.0: irq 102 for MSI/MSI-X [ 9.075949] mpt2sas 0000:82:00.0: irq 103 for MSI/MSI-X [ 9.075996] mpt2sas 0000:82:00.0: irq 104 for MSI/MSI-X [ 9.076019] mpt2sas 0000:82:00.0: irq 105 for MSI/MSI-X [ 9.076042] mpt2sas 0000:82:00.0: irq 106 for MSI/MSI-X [ 9.076064] mpt2sas 0000:82:00.0: irq 107 for MSI/MSI-X [ 9.076087] mpt2sas 0000:82:00.0: irq 108 for MSI/MSI-X [ 9.076110] mpt2sas 0000:82:00.0: irq 109 for MSI/MSI-X [ 9.076132] mpt2sas 0000:82:00.0: irq 110 for MSI/MSI-X [ 9.076155] mpt2sas 0000:82:00.0: irq 111 for MSI/MSI-X [ 9.076181] mpt2sas 0000:82:00.0: irq 112 for MSI/MSI-X [ 9.076204] mpt2sas 0000:82:00.0: irq 113 for MSI/MSI-X [ 9.076226] mpt2sas 0000:82:00.0: irq 114 for MSI/MSI-X [ 9.076249] mpt2sas 0000:82:00.0: irq 115 for MSI/MSI-X [ 9.076370] mpt2sas1-msix0: PCI-MSI-X enabled: IRQ 100 [ 9.076371] mpt2sas1-msix1: PCI-MSI-X enabled: IRQ 101 [ 9.076371] mpt2sas1-msix2: PCI-MSI-X enabled: IRQ 102 [ 9.076372] mpt2sas1-msix3: PCI-MSI-X enabled: IRQ 103 [ 9.076372] mpt2sas1-msix4: PCI-MSI-X enabled: IRQ 104 [ 9.076372] mpt2sas1-msix5: PCI-MSI-X enabled: IRQ 105 [ 9.076373] mpt2sas1-msix6: PCI-MSI-X enabled: IRQ 106 [ 9.076373] mpt2sas1-msix7: PCI-MSI-X enabled: IRQ 107 [ 9.076374] mpt2sas1-msix8: PCI-MSI-X enabled: IRQ 108 [ 9.076374] mpt2sas1-msix9: PCI-MSI-X enabled: IRQ 109 [ 9.076375] mpt2sas1-msix10: PCI-MSI-X enabled: IRQ 110 [ 9.076375] mpt2sas1-msix11: PCI-MSI-X enabled: IRQ 111 [ 9.076375] mpt2sas1-msix12: PCI-MSI-X enabled: IRQ 112 [ 9.076376] mpt2sas1-msix13: PCI-MSI-X enabled: IRQ 113 [ 9.076376] mpt2sas1-msix14: PCI-MSI-X enabled: IRQ 114 [ 9.076377] mpt2sas1-msix15: PCI-MSI-X enabled: IRQ 115 [ 9.076378] mpt2sas_cm1: iomem(0x00000000ec140000), mapped(0xffffc90003b20000), size(65536) [ 9.076379] mpt2sas_cm1: ioport(0x000000000000c000), size(256) [ 9.092893] ata1: SATA link down (SStatus 0 SControl 300) [ 9.092967] ata4: SATA link down (SStatus 0 SControl 300) [ 9.093042] ata2: SATA link down (SStatus 0 SControl 300) [ 9.093117] ata6: SATA link down (SStatus 0 SControl 300) [ 9.093874] ata5: SATA link down (SStatus 0 SControl 300) [ 9.096906] ata3: SATA link down (SStatus 0 SControl 300) [ 9.106152] [TTM] Zone kernel: Available graphics memory: 16255612 kiB [ 9.106153] [TTM] Zone dma32: Available graphics memory: 2097152 kiB [ 9.106154] [TTM] Initializing pool allocator [ 9.106166] [TTM] Initializing DMA pool allocator [ 9.117818] igb 0000:02:00.3: added PHC on eth3 [ 9.117819] igb 0000:02:00.3: Intel(R) Gigabit Ethernet Network Connection [ 9.117821] igb 0000:02:00.3: eth3: (PCIe:5.0Gb/s:Width x4) 00:1e:67:6b:23:2c [ 9.117893] igb 0000:02:00.3: eth3: PBA No: 100000-000 [ 9.117895] igb 0000:02:00.3: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s) [ 9.130825] mpt2sas_cm1: sending message unit reset !! [ 9.132353] mpt2sas_cm1: message unit reset: SUCCESS [ 9.144987] fbcon: mgadrmfb (fb0) is primary device [ 9.209759] mpt2sas_cm1: Allocated physical memory: size(18690 kB) [ 9.209760] mpt2sas_cm1: Current Controller Queue Depth(10104),Max Controller Queue Depth(10240) [ 9.209761] mpt2sas_cm1: Scatter Gather Elements per IO(128) [ 9.214334] scsi 0:0:1:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.214353] scsi 0:0:1:0: SSP: handle(0x000b), sas_addr(0x5000c5008429d8b1), phy(9), device_name(0x5000c5008429d8b0) [ 9.214356] scsi 0:0:1:0: SSP: enclosure_logical_id(0x500304800050553f), slot(1) [ 9.214360] scsi 0:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.255479] mpt2sas_cm1: LSISAS2308: FWVersion(20.00.07.00), ChipRevision(0x05), BiosVersion(07.39.00.00) [ 9.255481] mpt2sas_cm1: Protocol=( [ 9.255482] Initiator [ 9.255483] ,Target [ 9.255483] ), [ 9.255484] Capabilities=( [ 9.255485] TLR [ 9.255485] ,EEDP [ 9.255486] ,Snapshot Buffer [ 9.255486] ,Diag Trace Buffer [ 9.255487] ,Task Set Full [ 9.255488] ,NCQ [ 9.255488] ) [ 9.255542] scsi host8: Fusion MPT SAS Host [ 9.256230] mpt2sas_cm1: sending port enable !! [ 9.256637] scsi 0:0:2:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.256642] scsi 0:0:2:0: SSP: handle(0x000c), sas_addr(0x5000c5008429f0a5), phy(10), device_name(0x5000c5008429f0a4) [ 9.256644] scsi 0:0:2:0: SSP: enclosure_logical_id(0x500304800050553f), slot(2) [ 9.256647] scsi 0:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.259528] mpt2sas_cm1: host_add: handle(0x0001), sas_addr(0x500605b00a6d91d0), phys(8) [ 9.262229] mpt2sas_cm1: expander_add: handle(0x0009), parent(0x0001), sas_addr(0x50030480005270bf), phys(38) [ 9.275407] scsi 0:0:3:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.275427] scsi 0:0:3:0: SSP: handle(0x000d), sas_addr(0x5000c5008429ea75), phy(11), device_name(0x5000c5008429ea74) [ 9.275430] scsi 0:0:3:0: SSP: enclosure_logical_id(0x500304800050553f), slot(3) [ 9.275433] scsi 0:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.282575] mpt2sas_cm1: expander_add: handle(0x0020), parent(0x0002), sas_addr(0x50030480004053bf), phys(38) [ 9.294247] scsi 0:0:4:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.294253] scsi 0:0:4:0: SSP: handle(0x000e), sas_addr(0x5000c500842a9fad), phy(12), device_name(0x5000c500842a9fac) [ 9.294255] scsi 0:0:4:0: SSP: enclosure_logical_id(0x500304800050553f), slot(4) [ 9.294259] scsi 0:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.310853] mpt2sas_cm1: port enable: SUCCESS [ 9.312481] scsi 8:0:0:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.312501] scsi 8:0:0:0: SSP: handle(0x000a), sas_addr(0x5000c500842a08ba), phy(8), device_name(0x5000c500842a08b8) [ 9.312504] scsi 8:0:0:0: SSP: enclosure_logical_id(0x500304800052703f), slot(0) [ 9.312508] scsi 8:0:0:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.312532] scsi 0:0:5:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.312538] scsi 0:0:5:0: SSP: handle(0x000f), sas_addr(0x5000c500842a058d), phy(13), device_name(0x5000c500842a058c) [ 9.312541] scsi 0:0:5:0: SSP: enclosure_logical_id(0x500304800050553f), slot(5) [ 9.312544] scsi 0:0:5:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.327705] scsi 0:0:6:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.327711] scsi 0:0:6:0: SSP: handle(0x0010), sas_addr(0x5000c500842a4405), phy(14), device_name(0x5000c500842a4404) [ 9.327714] scsi 0:0:6:0: SSP: enclosure_logical_id(0x500304800050553f), slot(6) [ 9.327717] scsi 0:0:6:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.327753] scsi 8:0:1:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.327761] scsi 8:0:1:0: SSP: handle(0x000b), sas_addr(0x5000c500842a6922), phy(9), device_name(0x5000c500842a6920) [ 9.327764] scsi 8:0:1:0: SSP: enclosure_logical_id(0x500304800052703f), slot(1) [ 9.327769] scsi 8:0:1:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.351646] scsi 0:0:7:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.351652] scsi 0:0:7:0: SSP: handle(0x0011), sas_addr(0x5000c500842a52dd), phy(15), device_name(0x5000c500842a52dc) [ 9.351654] scsi 0:0:7:0: SSP: enclosure_logical_id(0x500304800050553f), slot(7) [ 9.351657] scsi 0:0:7:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.351846] scsi 8:0:2:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.351853] scsi 8:0:2:0: SSP: handle(0x000c), sas_addr(0x5000c500842a6ad6), phy(10), device_name(0x5000c500842a6ad4) [ 9.351856] scsi 8:0:2:0: SSP: enclosure_logical_id(0x500304800052703f), slot(2) [ 9.351859] scsi 8:0:2:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.380737] Console: switching to colour frame buffer device 128x48 [ 9.419522] mgag200 0000:09:00.0: fb0: mgadrmfb frame buffer device [ 9.447903] [drm] Initialized mgag200 1.0.0 20110418 for 0000:09:00.0 on minor 0 [ 9.449533] scsi 0:0:8:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.449539] scsi 0:0:8:0: SSP: handle(0x0012), sas_addr(0x5000c500842a9f59), phy(16), device_name(0x5000c500842a9f58) [ 9.449541] scsi 0:0:8:0: SSP: enclosure_logical_id(0x500304800050553f), slot(8) [ 9.449544] scsi 0:0:8:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.449685] scsi 8:0:3:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.449692] scsi 8:0:3:0: SSP: handle(0x000d), sas_addr(0x5000c5008429f8ee), phy(11), device_name(0x5000c5008429f8ec) [ 9.449694] scsi 8:0:3:0: SSP: enclosure_logical_id(0x500304800052703f), slot(3) [ 9.449697] scsi 8:0:3:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.542320] scsi 8:0:4:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.542324] scsi 8:0:4:0: SSP: handle(0x000e), sas_addr(0x5000c500842a486e), phy(12), device_name(0x5000c500842a486c) [ 9.542326] scsi 8:0:4:0: SSP: enclosure_logical_id(0x500304800052703f), slot(4) [ 9.542328] scsi 8:0:4:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 9.554383] scsi 0:0:9:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 9.554388] scsi 0:0:9:0: SSP: handle(0x0013), sas_addr(0x5000c500842a6bed), phy(17), device_name(0x5000c500842a6bec) [ 9.554390] scsi 0:0:9:0: SSP: enclosure_logical_id(0x500304800050553f), slot(9) [ 9.554393] scsi 0:0:9:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 10.393981] sas: phy-7:0 added to port-7:0, phy_mask:0x1 (5001e676b2329000) [ 10.394040] sas: phy-7:1 added to port-7:1, phy_mask:0x2 (5001e676b2329001) [ 10.394074] sas: DOING DISCOVERY on port 0, pid:228 [ 10.394125] sas: DONE DISCOVERY on port 0, pid:228, result:0 [ 10.394140] sas: DOING DISCOVERY on port 1, pid:228 [ 10.394190] sas: DONE DISCOVERY on port 1, pid:228, result:0 [ 10.394204] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 10.394288] sas: ata7: end_device-7:0: dev error handler [ 10.554764] ata7.00: ATA-8: ST500NM0011, SN02, max UDMA/133 [ 10.554767] ata7.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 10.555664] ata7.00: configured for UDMA/133 [ 10.555746] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 10.566754] scsi 7:0:0:0: Direct-Access ATA ST500NM0011 SN02 PQ: 0 ANSI: 5 [ 14.112958] scsi 0:0:10:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 14.114082] scsi 8:0:5:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 14.114089] scsi 8:0:5:0: SSP: handle(0x000f), sas_addr(0x5000c500842a692e), phy(13), device_name(0x5000c500842a692c) [ 14.114092] scsi 8:0:5:0: SSP: enclosure_logical_id(0x500304800052703f), slot(5) [ 14.114097] scsi 8:0:5:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 14.166068] scsi 0:0:10:0: SSP: handle(0x0014), sas_addr(0x5000c500842a4041), phy(18), device_name(0x5000c500842a4040) [ 14.178972] scsi 0:0:10:0: SSP: enclosure_logical_id(0x500304800050553f), slot(10) [ 14.188375] scsi 0:0:10:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 15.304001] mlx4_core 0000:04:00.0: Old device ETS support detected [ 15.311951] mlx4_core 0000:04:00.0: Consider upgrading device FW. [ 15.319708] mlx4_core 0000:04:00.0: DMFS high rate mode not supported [ 15.327838] mlx4_core: device is working in RoCE mode: Roce V1 [ 15.335275] mlx4_core: UD QP Gid type is: V1 [ 16.405820] mlx4_core 0000:04:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s [ 16.415674] mlx4_core 0000:04:00.0: PCIe link width is x8, device supports x8 [ 16.424751] mlx4_core 0000:04:00.0: irq 116 for MSI/MSI-X [ 16.424771] mlx4_core 0000:04:00.0: irq 117 for MSI/MSI-X [ 16.424790] mlx4_core 0000:04:00.0: irq 118 for MSI/MSI-X [ 16.424809] mlx4_core 0000:04:00.0: irq 119 for MSI/MSI-X [ 16.424848] mlx4_core 0000:04:00.0: irq 120 for MSI/MSI-X [ 16.424867] mlx4_core 0000:04:00.0: irq 121 for MSI/MSI-X [ 16.424890] mlx4_core 0000:04:00.0: irq 122 for MSI/MSI-X [ 16.424910] mlx4_core 0000:04:00.0: irq 123 for MSI/MSI-X [ 16.424929] mlx4_core 0000:04:00.0: irq 124 for MSI/MSI-X [ 16.424948] mlx4_core 0000:04:00.0: irq 125 for MSI/MSI-X [ 16.424969] mlx4_core 0000:04:00.0: irq 126 for MSI/MSI-X [ 16.424988] mlx4_core 0000:04:00.0: irq 127 for MSI/MSI-X [ 16.425008] mlx4_core 0000:04:00.0: irq 128 for MSI/MSI-X [ 16.425033] mlx4_core 0000:04:00.0: irq 129 for MSI/MSI-X [ 16.425056] mlx4_core 0000:04:00.0: irq 130 for MSI/MSI-X [ 16.425075] mlx4_core 0000:04:00.0: irq 131 for MSI/MSI-X [ 16.425101] mlx4_core 0000:04:00.0: irq 132 for MSI/MSI-X [ 16.425120] mlx4_core 0000:04:00.0: irq 133 for MSI/MSI-X [ 16.425143] mlx4_core 0000:04:00.0: irq 134 for MSI/MSI-X [ 16.425163] mlx4_core 0000:04:00.0: irq 135 for MSI/MSI-X [ 16.425201] mlx4_core 0000:04:00.0: irq 136 for MSI/MSI-X [ 16.425221] mlx4_core 0000:04:00.0: irq 137 for MSI/MSI-X [ 16.425243] mlx4_core 0000:04:00.0: irq 138 for MSI/MSI-X [ 16.425263] mlx4_core 0000:04:00.0: irq 139 for MSI/MSI-X [ 16.425282] mlx4_core 0000:04:00.0: irq 140 for MSI/MSI-X [ 16.425301] mlx4_core 0000:04:00.0: irq 141 for MSI/MSI-X [ 16.425320] mlx4_core 0000:04:00.0: irq 142 for MSI/MSI-X [ 16.425340] mlx4_core 0000:04:00.0: irq 143 for MSI/MSI-X [ 16.425359] mlx4_core 0000:04:00.0: irq 144 for MSI/MSI-X [ 16.425379] mlx4_core 0000:04:00.0: irq 145 for MSI/MSI-X [ 16.425401] mlx4_core 0000:04:00.0: irq 146 for MSI/MSI-X [ 16.425421] mlx4_core 0000:04:00.0: irq 147 for MSI/MSI-X [ 16.425440] mlx4_core 0000:04:00.0: irq 148 for MSI/MSI-X [ 16.425459] mlx4_core 0000:04:00.0: irq 149 for MSI/MSI-X [ 16.425478] mlx4_core 0000:04:00.0: irq 150 for MSI/MSI-X [ 16.425498] mlx4_core 0000:04:00.0: irq 151 for MSI/MSI-X [ 16.425536] mlx4_core 0000:04:00.0: irq 152 for MSI/MSI-X [ 16.425556] mlx4_core 0000:04:00.0: irq 153 for MSI/MSI-X [ 16.425579] mlx4_core 0000:04:00.0: irq 154 for MSI/MSI-X [ 16.425598] mlx4_core 0000:04:00.0: irq 155 for MSI/MSI-X [ 16.425617] mlx4_core 0000:04:00.0: irq 156 for MSI/MSI-X [ 16.425637] mlx4_core 0000:04:00.0: irq 157 for MSI/MSI-X [ 16.425656] mlx4_core 0000:04:00.0: irq 158 for MSI/MSI-X [ 16.425676] mlx4_core 0000:04:00.0: irq 159 for MSI/MSI-X [ 16.425695] mlx4_core 0000:04:00.0: irq 160 for MSI/MSI-X [ 16.425714] mlx4_core 0000:04:00.0: irq 161 for MSI/MSI-X [ 16.425737] mlx4_core 0000:04:00.0: irq 162 for MSI/MSI-X [ 16.425756] mlx4_core 0000:04:00.0: irq 163 for MSI/MSI-X [ 16.425776] mlx4_core 0000:04:00.0: irq 164 for MSI/MSI-X [ 16.425795] mlx4_core 0000:04:00.0: irq 165 for MSI/MSI-X [ 16.425814] mlx4_core 0000:04:00.0: irq 166 for MSI/MSI-X [ 16.425833] mlx4_core 0000:04:00.0: irq 167 for MSI/MSI-X [ 16.425876] mlx4_core 0000:04:00.0: irq 168 for MSI/MSI-X [ 16.425895] mlx4_core 0000:04:00.0: irq 169 for MSI/MSI-X [ 16.425918] mlx4_core 0000:04:00.0: irq 170 for MSI/MSI-X [ 16.425937] mlx4_core 0000:04:00.0: irq 171 for MSI/MSI-X [ 16.425957] mlx4_core 0000:04:00.0: irq 172 for MSI/MSI-X [ 16.425976] mlx4_core 0000:04:00.0: irq 173 for MSI/MSI-X [ 16.425995] mlx4_core 0000:04:00.0: irq 174 for MSI/MSI-X [ 16.426015] mlx4_core 0000:04:00.0: irq 175 for MSI/MSI-X [ 16.426037] mlx4_core 0000:04:00.0: irq 176 for MSI/MSI-X [ 16.426056] mlx4_core 0000:04:00.0: irq 177 for MSI/MSI-X [ 16.426088] mlx4_core 0000:04:00.0: irq 178 for MSI/MSI-X [ 16.426108] mlx4_core 0000:04:00.0: irq 179 for MSI/MSI-X [ 16.426127] mlx4_core 0000:04:00.0: irq 180 for MSI/MSI-X [ 16.544188] sas: Enter sas_scsi_recover_host busy: 0 failed: 0 [ 16.544246] sas: ata7: end_device-7:0: dev error handler [ 16.544258] sas: ata8: end_device-7:1: dev error handler [ 16.685370] mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.2-1.0.0 [ 16.685492] scsi 0:0:11:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 16.685500] scsi 0:0:11:0: SSP: handle(0x0015), sas_addr(0x5000c5008429e625), phy(19), device_name(0x5000c5008429e624) [ 16.685503] scsi 0:0:11:0: SSP: enclosure_logical_id(0x500304800050553f), slot(11) [ 16.685520] scsi 0:0:11:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 16.686011] scsi 8:0:6:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 16.686019] scsi 8:0:6:0: SSP: handle(0x0010), sas_addr(0x5000c5008429f81e), phy(14), device_name(0x5000c5008429f81c) [ 16.686021] scsi 8:0:6:0: SSP: enclosure_logical_id(0x500304800052703f), slot(6) [ 16.686025] scsi 8:0:6:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 16.704824] ata8.00: ATA-8: ST500NM0011, SN02, max UDMA/133 [ 16.704826] ata8.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 16.705562] ata8.00: configured for UDMA/133 [ 16.705586] sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1 [ 16.717762] scsi 7:0:1:0: Direct-Access ATA ST500NM0011 SN02 PQ: 0 ANSI: 5 [ 16.819202] mlx4_ib_add: counter index 0 for port 1 allocated 0 [ 16.827861] mlx4_ib_add: counter index 1 for port 2 allocated 0 [ 16.851762] scsi 8:0:7:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 16.851794] scsi 0:0:12:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 16.851799] scsi 0:0:12:0: SSP: handle(0x0016), sas_addr(0x5000c500842a1481), phy(20), device_name(0x5000c500842a1480) [ 16.851801] scsi 0:0:12:0: SSP: enclosure_logical_id(0x500304800050553f), slot(12) [ 16.851803] scsi 0:0:12:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 16.909904] scsi 8:0:7:0: SSP: handle(0x0011), sas_addr(0x5000c500842a097a), phy(15), device_name(0x5000c500842a0978) [ 16.923247] scsi 8:0:7:0: SSP: enclosure_logical_id(0x500304800052703f), slot(7) [ 16.932944] scsi 8:0:7:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 16.950065] sd 7:0:0:0: [sda] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 16.950093] sd 7:0:0:0: [sda] Write Protect is off [ 16.950095] sd 7:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 16.950106] sd 7:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 16.950145] sd 7:0:1:0: [sdb] 976773168 512-byte logical blocks: (500 GB/465 GiB) [ 16.950282] sd 7:0:1:0: [sdb] Write Protect is off [ 16.950283] sd 7:0:1:0: [sdb] Mode Sense: 00 3a 00 00 [ 16.950308] sd 7:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 16.960524] sdb: [ 16.960808] sd 7:0:1:0: [sdb] Attached SCSI disk [ 16.963056] sda: sda1 sda2 [ 16.963483] sd 7:0:0:0: [sda] Attached SCSI disk [ 17.056320] scsi 0:0:13:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.065893] scsi 8:0:8:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.065901] scsi 8:0:8:0: SSP: handle(0x0012), sas_addr(0x5000c500842a6e6e), phy(16), device_name(0x5000c500842a6e6c) [ 17.065904] scsi 8:0:8:0: SSP: enclosure_logical_id(0x500304800052703f), slot(8) [ 17.065908] scsi 8:0:8:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.110000] scsi 0:0:13:0: SSP: handle(0x0017), sas_addr(0x5000c5008429ecf9), phy(21), device_name(0x5000c5008429ecf8) [ 17.122976] scsi 0:0:13:0: SSP: enclosure_logical_id(0x500304800050553f), slot(13) [ 17.132488] scsi 0:0:13:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.176670] scsi 0:0:14:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.177032] scsi 8:0:9:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.177037] scsi 8:0:9:0: SSP: handle(0x0013), sas_addr(0x5000c500842a24c6), phy(17), device_name(0x5000c500842a24c4) [ 17.177039] scsi 8:0:9:0: SSP: enclosure_logical_id(0x500304800052703f), slot(9) [ 17.177043] scsi 8:0:9:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.230353] scsi 0:0:14:0: SSP: handle(0x0018), sas_addr(0x5000c500842a5ea5), phy(22), device_name(0x5000c500842a5ea4) [ 17.243392] scsi 0:0:14:0: SSP: enclosure_logical_id(0x500304800050553f), slot(14) [ 17.252970] scsi 0:0:14:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.277749] scsi 8:0:10:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.281475] scsi 0:0:15:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.281479] scsi 0:0:15:0: SSP: handle(0x0019), sas_addr(0x5000c5008429fa19), phy(23), device_name(0x5000c5008429fa18) [ 17.281481] scsi 0:0:15:0: SSP: enclosure_logical_id(0x500304800050553f), slot(15) [ 17.281483] scsi 0:0:15:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.336634] scsi 8:0:10:0: SSP: handle(0x0014), sas_addr(0x5000c500842a5d7e), phy(18), device_name(0x5000c500842a5d7c) [ 17.350262] scsi 8:0:10:0: SSP: enclosure_logical_id(0x500304800052703f), slot(10) [ 17.360405] scsi 8:0:10:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.386653] scsi 8:0:11:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.397920] scsi 8:0:11:0: SSP: handle(0x0015), sas_addr(0x5000c5008429e7ee), phy(19), device_name(0x5000c5008429e7ec) [ 17.411906] scsi 8:0:11:0: SSP: enclosure_logical_id(0x500304800052703f), slot(11) [ 17.422405] scsi 8:0:11:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.438718] scsi 0:0:16:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.450481] scsi 0:0:16:0: SSP: handle(0x001a), sas_addr(0x5000c500842a6019), phy(24), device_name(0x5000c500842a6018) [ 17.464832] scsi 0:0:16:0: SSP: enclosure_logical_id(0x500304800050553f), slot(16) [ 17.475672] scsi 0:0:16:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.496807] scsi 8:0:12:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.508682] scsi 8:0:12:0: SSP: handle(0x0016), sas_addr(0x5000c5008427aa7a), phy(20), device_name(0x5000c5008427aa78) [ 17.523102] scsi 8:0:12:0: SSP: enclosure_logical_id(0x500304800052703f), slot(12) [ 17.533985] scsi 8:0:12:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.549741] scsi 0:0:17:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.549748] scsi 0:0:17:0: SSP: handle(0x001b), sas_addr(0x5000c5008429fb81), phy(25), device_name(0x5000c5008429fb80) [ 17.549751] scsi 0:0:17:0: SSP: enclosure_logical_id(0x500304800050553f), slot(17) [ 17.549754] scsi 0:0:17:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.628790] scsi 8:0:13:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.631506] scsi 0:0:18:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.631517] scsi 0:0:18:0: SSP: handle(0x001c), sas_addr(0x5000c500842a5d2d), phy(26), device_name(0x5000c500842a5d2c) [ 17.631519] scsi 0:0:18:0: SSP: enclosure_logical_id(0x500304800050553f), slot(18) [ 17.631522] scsi 0:0:18:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.689063] scsi 8:0:13:0: SSP: handle(0x0017), sas_addr(0x5000c5008429fb56), phy(21), device_name(0x5000c5008429fb54) [ 17.702882] scsi 8:0:13:0: SSP: enclosure_logical_id(0x500304800052703f), slot(13) [ 17.713256] scsi 8:0:13:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.737542] scsi 8:0:14:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.749035] scsi 8:0:14:0: SSP: handle(0x0018), sas_addr(0x5000c5008429f216), phy(22), device_name(0x5000c5008429f214) [ 17.763207] scsi 8:0:14:0: SSP: enclosure_logical_id(0x500304800052703f), slot(14) [ 17.773860] scsi 8:0:14:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.788565] scsi 0:0:19:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.800246] scsi 0:0:19:0: SSP: handle(0x001d), sas_addr(0x5000c500842a5f51), phy(27), device_name(0x5000c500842a5f50) [ 17.814515] scsi 0:0:19:0: SSP: enclosure_logical_id(0x500304800050553f), slot(19) [ 17.825284] scsi 0:0:19:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.846806] scsi 8:0:15:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.858819] scsi 8:0:15:0: SSP: handle(0x0019), sas_addr(0x5000c500842a681a), phy(23), device_name(0x5000c500842a6818) [ 17.873348] scsi 8:0:15:0: SSP: enclosure_logical_id(0x500304800052703f), slot(15) [ 17.884398] scsi 8:0:15:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.899786] scsi 0:0:20:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.911681] scsi 0:0:20:0: SSP: handle(0x001e), sas_addr(0x5000c500842a03f5), phy(28), device_name(0x5000c500842a03f4) [ 17.926144] scsi 0:0:20:0: SSP: enclosure_logical_id(0x500304800050553f), slot(20) [ 17.937112] scsi 0:0:20:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 17.980772] scsi 8:0:16:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 17.992839] scsi 8:0:16:0: SSP: handle(0x001a), sas_addr(0x5000c500842a446e), phy(24), device_name(0x5000c500842a446c) [ 18.007368] scsi 8:0:16:0: SSP: enclosure_logical_id(0x500304800052703f), slot(16) [ 18.018464] scsi 8:0:16:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.033490] scsi 0:0:21:0: Enclosure LSI SAS2X36 0e0b PQ: 0 ANSI: 5 [ 18.045213] scsi 0:0:21:0: set ignore_delay_remove for handle(0x001f) [ 18.054738] scsi 0:0:21:0: SES: handle(0x001f), sas_addr(0x500304800050553d), phy(36), device_name(0x0000000000000000) [ 18.069046] scsi 0:0:21:0: SES: enclosure_logical_id(0x500304800050553f), slot(21) [ 18.079844] scsi 0:0:21:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 18.101784] scsi 8:0:17:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.113815] scsi 8:0:17:0: SSP: handle(0x001b), sas_addr(0x5000c500842a6afa), phy(25), device_name(0x5000c500842a6af8) [ 18.128324] scsi 8:0:17:0: SSP: enclosure_logical_id(0x500304800052703f), slot(17) [ 18.139411] scsi 8:0:17:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.154639] scsi 0:0:22:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.166377] scsi 0:0:22:0: SSP: handle(0x0021), sas_addr(0x5000c500842a64cd), phy(12), device_name(0x5000c500842a64cc) [ 18.180702] scsi 0:0:22:0: SSP: enclosure_logical_id(0x500304800040543f), slot(0) [ 18.191417] scsi 0:0:22:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.212835] scsi 8:0:18:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.224862] scsi 8:0:18:0: SSP: handle(0x001c), sas_addr(0x5000c500842a5a5a), phy(26), device_name(0x5000c500842a5a58) [ 18.239453] scsi 8:0:18:0: SSP: enclosure_logical_id(0x500304800052703f), slot(18) [ 18.250465] scsi 8:0:18:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.265776] scsi 0:0:23:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.277727] scsi 0:0:23:0: SSP: handle(0x0022), sas_addr(0x5000c500842a15e1), phy(13), device_name(0x5000c500842a15e0) [ 18.292256] scsi 0:0:23:0: SSP: enclosure_logical_id(0x500304800040543f), slot(1) [ 18.303155] scsi 0:0:23:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.324864] scsi 8:0:19:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.336960] scsi 8:0:19:0: SSP: handle(0x001d), sas_addr(0x5000c5008429f43e), phy(27), device_name(0x5000c5008429f43c) [ 18.351592] scsi 8:0:19:0: SSP: enclosure_logical_id(0x500304800052703f), slot(19) [ 18.362648] scsi 8:0:19:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.378823] scsi 0:0:24:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.390812] scsi 0:0:24:0: SSP: handle(0x0023), sas_addr(0x5000c500842a0b71), phy(14), device_name(0x5000c500842a0b70) [ 18.405373] scsi 0:0:24:0: SSP: enclosure_logical_id(0x500304800040543f), slot(2) [ 18.416324] scsi 0:0:24:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.438396] scsi 8:0:20:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.450444] scsi 8:0:20:0: SSP: handle(0x001e), sas_addr(0x5000c500842a189e), phy(28), device_name(0x5000c500842a189c) [ 18.465053] scsi 8:0:20:0: SSP: enclosure_logical_id(0x500304800052703f), slot(20) [ 18.476100] scsi 8:0:20:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.491788] scsi 0:0:25:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.503764] scsi 0:0:25:0: SSP: handle(0x0024), sas_addr(0x5000c5008429e331), phy(15), device_name(0x5000c5008429e330) [ 18.518325] scsi 0:0:25:0: SSP: enclosure_logical_id(0x500304800040543f), slot(3) [ 18.529275] scsi 0:0:25:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.550645] scsi 8:0:21:0: Enclosure LSI SAS2X36 0e0b PQ: 0 ANSI: 5 [ 18.550651] scsi 8:0:21:0: set ignore_delay_remove for handle(0x001f) [ 18.550655] scsi 8:0:21:0: SES: handle(0x001f), sas_addr(0x50030480005270bd), phy(36), device_name(0x0000000000000000) [ 18.550658] scsi 8:0:21:0: SES: enclosure_logical_id(0x500304800052703f), slot(21) [ 18.550661] scsi 8:0:21:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 18.630494] scsi 0:0:26:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.641206] scsi 0:0:26:0: SSP: handle(0x0025), sas_addr(0x5000c500842a52fd), phy(16), device_name(0x5000c500842a52fc) [ 18.654743] scsi 0:0:26:0: SSP: enclosure_logical_id(0x500304800040543f), slot(4) [ 18.664676] scsi 0:0:26:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.682781] scsi 8:0:22:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.694894] scsi 8:0:22:0: SSP: handle(0x0021), sas_addr(0x5000c500842a477e), phy(12), device_name(0x5000c500842a477c) [ 18.709505] scsi 8:0:22:0: SSP: enclosure_logical_id(0x500304800040533f), slot(0) [ 18.720446] scsi 8:0:22:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.738768] scsi 0:0:27:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.751170] scsi 0:0:27:0: SSP: handle(0x0026), sas_addr(0x5000c5008429f051), phy(17), device_name(0x5000c5008429f050) [ 18.751173] scsi 0:0:27:0: SSP: enclosure_logical_id(0x500304800040543f), slot(5) [ 18.751177] scsi 0:0:27:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.811857] scsi 8:0:23:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.823934] scsi 8:0:23:0: SSP: handle(0x0022), sas_addr(0x5000c500842a5cca), phy(13), device_name(0x5000c500842a5cc8) [ 18.838653] scsi 8:0:23:0: SSP: enclosure_logical_id(0x500304800040533f), slot(1) [ 18.849642] scsi 8:0:23:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.868684] scsi 0:0:28:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.880465] scsi 0:0:28:0: SSP: handle(0x0027), sas_addr(0x5000c500842a19d5), phy(18), device_name(0x5000c500842a19d4) [ 18.894807] scsi 0:0:28:0: SSP: enclosure_logical_id(0x500304800040543f), slot(6) [ 18.905549] scsi 0:0:28:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.920848] scsi 8:0:24:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.932924] scsi 8:0:24:0: SSP: handle(0x0023), sas_addr(0x5000c5008429ec5a), phy(14), device_name(0x5000c5008429ec58) [ 18.947500] scsi 8:0:24:0: SSP: enclosure_logical_id(0x500304800040533f), slot(2) [ 18.958462] scsi 8:0:24:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 18.979827] scsi 0:0:29:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 18.991814] scsi 0:0:29:0: SSP: handle(0x0028), sas_addr(0x5000c500842a280d), phy(19), device_name(0x5000c500842a280c) [ 19.006340] scsi 0:0:29:0: SSP: enclosure_logical_id(0x500304800040543f), slot(7) [ 19.017272] scsi 0:0:29:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.032877] scsi 8:0:25:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.044964] scsi 8:0:25:0: SSP: handle(0x0024), sas_addr(0x5000c500842a1342), phy(15), device_name(0x5000c500842a1340) [ 19.059586] scsi 8:0:25:0: SSP: enclosure_logical_id(0x500304800040533f), slot(3) [ 19.070583] scsi 8:0:25:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.091788] scsi 0:0:30:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.103553] scsi 0:0:30:0: SSP: handle(0x0029), sas_addr(0x5000c500842a6c09), phy(20), device_name(0x5000c500842a6c08) [ 19.117897] scsi 0:0:30:0: SSP: enclosure_logical_id(0x500304800040543f), slot(8) [ 19.128646] scsi 0:0:30:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.143790] scsi 8:0:26:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.155897] scsi 8:0:26:0: SSP: handle(0x0025), sas_addr(0x5000c500842a5b82), phy(16), device_name(0x5000c500842a5b80) [ 19.170488] scsi 8:0:26:0: SSP: enclosure_logical_id(0x500304800040533f), slot(4) [ 19.181479] scsi 8:0:26:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.205808] scsi 0:0:31:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.208801] scsi 8:0:27:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.208807] scsi 8:0:27:0: SSP: handle(0x0026), sas_addr(0x5000c5008429e56e), phy(17), device_name(0x5000c5008429e56c) [ 19.208810] scsi 8:0:27:0: SSP: enclosure_logical_id(0x500304800040533f), slot(5) [ 19.208813] scsi 8:0:27:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.265745] scsi 0:0:31:0: SSP: handle(0x002a), sas_addr(0x5000c50084298f01), phy(21), device_name(0x5000c50084298f00) [ 19.279565] scsi 0:0:31:0: SSP: enclosure_logical_id(0x500304800040543f), slot(9) [ 19.289784] scsi 0:0:31:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.316746] scsi 0:0:32:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.319918] scsi 8:0:28:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.319924] scsi 8:0:28:0: SSP: handle(0x0027), sas_addr(0x5000c5008429d276), phy(18), device_name(0x5000c5008429d274) [ 19.319926] scsi 8:0:28:0: SSP: enclosure_logical_id(0x500304800040533f), slot(6) [ 19.319930] scsi 8:0:28:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.375817] scsi 0:0:32:0: SSP: handle(0x002b), sas_addr(0x5000c500842a0d41), phy(22), device_name(0x5000c500842a0d40) [ 19.375819] scsi 0:0:32:0: SSP: enclosure_logical_id(0x500304800040543f), slot(10) [ 19.375822] scsi 0:0:32:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.444890] scsi 8:0:29:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.444971] scsi 0:0:33:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.444977] scsi 0:0:33:0: SSP: handle(0x002c), sas_addr(0x5000c500842a02c1), phy(23), device_name(0x5000c500842a02c0) [ 19.444980] scsi 0:0:33:0: SSP: enclosure_logical_id(0x500304800040543f), slot(11) [ 19.444983] scsi 0:0:33:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.505493] scsi 8:0:29:0: SSP: handle(0x0028), sas_addr(0x5000c500842a36f2), phy(19), device_name(0x5000c500842a36f0) [ 19.519302] scsi 8:0:29:0: SSP: enclosure_logical_id(0x500304800040533f), slot(7) [ 19.529578] scsi 8:0:29:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.553811] scsi 0:0:34:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.553837] scsi 0:0:34:0: SSP: handle(0x002d), sas_addr(0x5000c500842a1fd5), phy(24), device_name(0x5000c500842a1fd4) [ 19.553839] scsi 0:0:34:0: SSP: enclosure_logical_id(0x500304800040543f), slot(12) [ 19.553842] scsi 0:0:34:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.556518] scsi 8:0:30:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.556525] scsi 8:0:30:0: SSP: handle(0x0029), sas_addr(0x5000c500842a5a2e), phy(20), device_name(0x5000c500842a5a2c) [ 19.556540] scsi 8:0:30:0: SSP: enclosure_logical_id(0x500304800040533f), slot(8) [ 19.556543] scsi 8:0:30:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.684004] scsi 0:0:35:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.684078] scsi 8:0:31:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.684083] scsi 8:0:31:0: SSP: handle(0x002a), sas_addr(0x5000c5008429f156), phy(21), device_name(0x5000c5008429f154) [ 19.684085] scsi 8:0:31:0: SSP: enclosure_logical_id(0x500304800040533f), slot(9) [ 19.684089] scsi 8:0:31:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.740121] scsi 0:0:35:0: SSP: handle(0x002e), sas_addr(0x5000c500842a1ef9), phy(25), device_name(0x5000c500842a1ef8) [ 19.753335] scsi 0:0:35:0: SSP: enclosure_logical_id(0x500304800040543f), slot(13) [ 19.763104] scsi 0:0:35:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.807837] scsi 8:0:32:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.811661] scsi 0:0:36:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.811693] scsi 0:0:36:0: SSP: handle(0x002f), sas_addr(0x5000c500842a31d9), phy(26), device_name(0x5000c500842a31d8) [ 19.811694] scsi 0:0:36:0: SSP: enclosure_logical_id(0x500304800040543f), slot(14) [ 19.811696] scsi 0:0:36:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.867122] scsi 8:0:32:0: SSP: handle(0x002b), sas_addr(0x5000c5008429f4c6), phy(22), device_name(0x5000c5008429f4c4) [ 19.880797] scsi 8:0:32:0: SSP: enclosure_logical_id(0x500304800040533f), slot(10) [ 19.890938] scsi 8:0:32:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.918771] scsi 8:0:33:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.919154] scsi 0:0:37:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 19.919176] scsi 0:0:37:0: SSP: handle(0x0030), sas_addr(0x5000c5008429eff9), phy(27), device_name(0x5000c5008429eff8) [ 19.919179] scsi 0:0:37:0: SSP: enclosure_logical_id(0x500304800040543f), slot(15) [ 19.919182] scsi 0:0:37:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 19.976636] scsi 8:0:33:0: SSP: handle(0x002c), sas_addr(0x5000c500842a64a6), phy(23), device_name(0x5000c500842a64a4) [ 19.990085] scsi 8:0:33:0: SSP: enclosure_logical_id(0x500304800040533f), slot(11) [ 20.000058] scsi 8:0:33:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.023845] scsi 0:0:38:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.026582] scsi 8:0:34:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.026601] scsi 8:0:34:0: SSP: handle(0x002d), sas_addr(0x5000c5008429f2a6), phy(24), device_name(0x5000c5008429f2a4) [ 20.026603] scsi 8:0:34:0: SSP: enclosure_logical_id(0x500304800040533f), slot(12) [ 20.026606] scsi 8:0:34:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.083851] scsi 0:0:38:0: SSP: handle(0x0031), sas_addr(0x5000c500842a5025), phy(28), device_name(0x5000c500842a5024) [ 20.097627] scsi 0:0:38:0: SSP: enclosure_logical_id(0x500304800040543f), slot(16) [ 20.107866] scsi 0:0:38:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.135610] scsi 0:0:39:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.135688] scsi 8:0:35:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.135694] scsi 8:0:35:0: SSP: handle(0x002e), sas_addr(0x5000c500842a09d2), phy(25), device_name(0x5000c500842a09d0) [ 20.135710] scsi 8:0:35:0: SSP: enclosure_logical_id(0x500304800040533f), slot(13) [ 20.135714] scsi 8:0:35:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.194123] scsi 0:0:39:0: SSP: handle(0x0032), sas_addr(0x5000c500842a346d), phy(29), device_name(0x5000c500842a346c) [ 20.194128] scsi 0:0:39:0: SSP: enclosure_logical_id(0x500304800040543f), slot(17) [ 20.194134] scsi 0:0:39:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.259936] scsi 8:0:36:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.263763] scsi 0:0:40:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.263769] scsi 0:0:40:0: SSP: handle(0x0033), sas_addr(0x5000c500842a3955), phy(30), device_name(0x5000c500842a3954) [ 20.263771] scsi 0:0:40:0: SSP: enclosure_logical_id(0x500304800040543f), slot(18) [ 20.263774] scsi 0:0:40:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.319669] scsi 8:0:36:0: SSP: handle(0x002f), sas_addr(0x5000c500842a0c9e), phy(26), device_name(0x5000c500842a0c9c) [ 20.333419] scsi 8:0:36:0: SSP: enclosure_logical_id(0x500304800040533f), slot(14) [ 20.343629] scsi 8:0:36:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.367804] scsi 0:0:41:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.370798] scsi 8:0:37:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.370804] scsi 8:0:37:0: SSP: handle(0x0030), sas_addr(0x5000c500842a0c22), phy(27), device_name(0x5000c500842a0c20) [ 20.370806] scsi 8:0:37:0: SSP: enclosure_logical_id(0x500304800040533f), slot(15) [ 20.370810] scsi 8:0:37:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.427725] scsi 0:0:41:0: SSP: handle(0x0034), sas_addr(0x5000c5008429f541), phy(31), device_name(0x5000c5008429f540) [ 20.427728] scsi 0:0:41:0: SSP: enclosure_logical_id(0x500304800040543f), slot(19) [ 20.427731] scsi 0:0:41:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.496767] scsi 0:0:42:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.499926] scsi 8:0:38:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.499932] scsi 8:0:38:0: SSP: handle(0x0031), sas_addr(0x5000c5008429f7ca), phy(28), device_name(0x5000c5008429f7c8) [ 20.499934] scsi 8:0:38:0: SSP: enclosure_logical_id(0x500304800040533f), slot(16) [ 20.499938] scsi 8:0:38:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.554476] scsi 0:0:42:0: SSP: handle(0x0035), sas_addr(0x5000c5008429acbd), phy(32), device_name(0x5000c5008429acbc) [ 20.567930] scsi 0:0:42:0: SSP: enclosure_logical_id(0x500304800040543f), slot(20) [ 20.577910] scsi 0:0:42:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.622866] scsi 8:0:39:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.625599] scsi 0:0:43:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.625606] scsi 0:0:43:0: SSP: handle(0x0036), sas_addr(0x5000c5008429d531), phy(33), device_name(0x5000c5008429d530) [ 20.625621] scsi 0:0:43:0: SSP: enclosure_logical_id(0x500304800040543f), slot(21) [ 20.625623] scsi 0:0:43:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.682768] scsi 8:0:39:0: SSP: handle(0x0032), sas_addr(0x5000c500842a06e6), phy(29), device_name(0x5000c500842a06e4) [ 20.696529] scsi 8:0:39:0: SSP: enclosure_logical_id(0x500304800040533f), slot(17) [ 20.706762] scsi 8:0:39:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.730812] scsi 0:0:44:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.733962] scsi 8:0:40:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.733969] scsi 8:0:40:0: SSP: handle(0x0033), sas_addr(0x5000c5008429f4fe), phy(30), device_name(0x5000c5008429f4fc) [ 20.733971] scsi 8:0:40:0: SSP: enclosure_logical_id(0x500304800040533f), slot(18) [ 20.733975] scsi 8:0:40:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.792228] scsi 0:0:44:0: SSP: handle(0x0037), sas_addr(0x5000c500842a1c49), phy(34), device_name(0x5000c500842a1c48) [ 20.792232] scsi 0:0:44:0: SSP: enclosure_logical_id(0x500304800040543f), slot(22) [ 20.792234] scsi 0:0:44:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.858769] scsi 0:0:45:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.862115] scsi 8:0:41:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.862122] scsi 8:0:41:0: SSP: handle(0x0034), sas_addr(0x5000c5008429fcfa), phy(31), device_name(0x5000c5008429fcf8) [ 20.862125] scsi 8:0:41:0: SSP: enclosure_logical_id(0x500304800040533f), slot(19) [ 20.862129] scsi 8:0:41:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.915552] scsi 0:0:45:0: SSP: handle(0x0038), sas_addr(0x5000c500842a4589), phy(35), device_name(0x5000c500842a4588) [ 20.928848] scsi 0:0:45:0: SSP: enclosure_logical_id(0x500304800040543f), slot(23) [ 20.938696] scsi 0:0:45:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 20.965364] scsi 0:0:46:0: Enclosure LSI SAS2X36 0e0b PQ: 0 ANSI: 5 [ 20.965413] scsi 8:0:42:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 20.965421] scsi 8:0:42:0: SSP: handle(0x0035), sas_addr(0x5000c500842a5c9e), phy(32), device_name(0x5000c500842a5c9c) [ 20.965424] scsi 8:0:42:0: SSP: enclosure_logical_id(0x500304800040533f), slot(20) [ 20.965428] scsi 8:0:42:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 21.022655] scsi 0:0:46:0: set ignore_delay_remove for handle(0x0039) [ 21.022658] scsi 0:0:46:0: SES: handle(0x0039), sas_addr(0x500304800040543d), phy(36), device_name(0x0000000000000000) [ 21.022660] scsi 0:0:46:0: SES: enclosure_logical_id(0x500304800040543f), slot(24) [ 21.022687] scsi 0:0:46:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 21.096895] scsi 8:0:43:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 21.100294] sd 0:0:0:0: [sdc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.100562] sd 0:0:1:0: [sdd] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.100858] sd 0:0:2:0: [sde] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.101103] sd 0:0:3:0: [sdf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.101364] sd 0:0:4:0: [sdg] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.101407] sd 0:0:0:0: [sdc] Write Protect is off [ 21.101409] sd 0:0:0:0: [sdc] Mode Sense: db 00 10 08 [ 21.101579] sd 0:0:5:0: [sdh] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.101652] sd 0:0:1:0: [sdd] Write Protect is off [ 21.101653] sd 0:0:1:0: [sdd] Mode Sense: db 00 10 08 [ 21.101755] sd 0:0:6:0: [sdi] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.101994] sd 0:0:2:0: [sde] Write Protect is off [ 21.101995] sd 0:0:2:0: [sde] Mode Sense: db 00 10 08 [ 21.102001] sd 0:0:0:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.102063] sd 0:0:7:0: [sdj] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.102224] sd 0:0:3:0: [sdf] Write Protect is off [ 21.102225] sd 0:0:3:0: [sdf] Mode Sense: db 00 10 08 [ 21.102259] sd 0:0:1:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.102313] sd 0:0:8:0: [sdk] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.102467] sd 0:0:4:0: [sdg] Write Protect is off [ 21.102468] sd 0:0:4:0: [sdg] Mode Sense: db 00 10 08 [ 21.102523] sd 0:0:9:0: [sdl] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.102587] sd 0:0:2:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.102636] sd 0:0:10:0: [sdm] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.102681] sd 0:0:5:0: [sdh] Write Protect is off [ 21.102682] sd 0:0:5:0: [sdh] Mode Sense: db 00 10 08 [ 21.102820] sd 0:0:3:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.102859] sd 0:0:11:0: [sdn] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.103005] sd 0:0:6:0: [sdi] Write Protect is off [ 21.103007] sd 0:0:6:0: [sdi] Mode Sense: db 00 10 08 [ 21.103070] sd 0:0:4:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.103130] sd 0:0:12:0: [sdo] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.103201] sd 0:0:7:0: [sdj] Write Protect is off [ 21.103202] sd 0:0:7:0: [sdj] Mode Sense: db 00 10 08 [ 21.103287] sd 0:0:5:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.103354] sd 0:0:13:0: [sdp] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.103448] sd 0:0:8:0: [sdk] Write Protect is off [ 21.103449] sd 0:0:8:0: [sdk] Mode Sense: db 00 10 08 [ 21.103543] sd 0:0:14:0: [sdq] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.103593] sd 0:0:6:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.103604] sd 0:0:9:0: [sdl] Write Protect is off [ 21.103606] sd 0:0:9:0: [sdl] Mode Sense: db 00 10 08 [ 21.103734] sd 0:0:10:0: [sdm] Write Protect is off [ 21.103736] sd 0:0:10:0: [sdm] Mode Sense: db 00 10 08 [ 21.103803] sd 0:0:7:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.103867] sd 0:0:15:0: [sdr] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.103977] sd 0:0:11:0: [sdn] Write Protect is off [ 21.103979] sd 0:0:11:0: [sdn] Mode Sense: db 00 10 08 [ 21.104058] sd 0:0:8:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.104080] sd 0:0:16:0: [sds] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.104213] sd 0:0:9:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.104254] sd 0:0:12:0: [sdo] Write Protect is off [ 21.104256] sd 0:0:12:0: [sdo] Mode Sense: db 00 10 08 [ 21.104308] sd 0:0:10:0: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.104316] sd 0:0:17:0: [sdt] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.104461] sd 0:0:13:0: [sdp] Write Protect is off [ 21.104462] sd 0:0:13:0: [sdp] Mode Sense: db 00 10 08 [ 21.104554] sd 0:0:11:0: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.104615] sd 0:0:18:0: [sdu] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.104642] sd 0:0:14:0: [sdq] Write Protect is off [ 21.104643] sd 0:0:14:0: [sdq] Mode Sense: db 00 10 08 [ 21.104815] sd 0:0:12:0: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.104981] sd 0:0:19:0: [sdv] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.104996] sd 0:0:15:0: [sdr] Write Protect is off [ 21.104997] sd 0:0:15:0: [sdr] Mode Sense: db 00 10 08 [ 21.105030] sd 0:0:13:0: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.105172] sd 0:0:16:0: [sds] Write Protect is off [ 21.105173] sd 0:0:16:0: [sds] Mode Sense: db 00 10 08 [ 21.105191] sd 0:0:20:0: [sdw] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.105216] sd 0:0:14:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.105416] sd 0:0:17:0: [sdt] Write Protect is off [ 21.105418] sd 0:0:17:0: [sdt] Mode Sense: db 00 10 08 [ 21.105467] sd 0:0:22:0: [sdx] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.105602] sd 0:0:15:0: [sdr] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.105689] sd 0:0:23:0: [sdy] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.105731] sd 0:0:18:0: [sdu] Write Protect is off [ 21.105732] sd 0:0:18:0: [sdu] Mode Sense: db 00 10 08 [ 21.105774] sd 0:0:16:0: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.105995] sd 0:0:17:0: [sdt] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.106019] sd 0:0:24:0: [sdz] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.106149] sd 0:0:19:0: [sdv] Write Protect is off [ 21.106150] sd 0:0:19:0: [sdv] Mode Sense: db 00 10 08 [ 21.106325] random: crng init done [ 21.106342] sd 0:0:25:0: [sdaa] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.106351] sd 0:0:20:0: [sdw] Write Protect is off [ 21.106353] sd 0:0:20:0: [sdw] Mode Sense: db 00 10 08 [ 21.106370] sd 0:0:18:0: [sdu] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.106566] sd 0:0:22:0: [sdx] Write Protect is off [ 21.106568] sd 0:0:22:0: [sdx] Mode Sense: db 00 10 08 [ 21.106665] sd 0:0:26:0: [sdab] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.106722] sd 0:0:19:0: [sdv] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.106824] sd 0:0:23:0: [sdy] Write Protect is off [ 21.106826] sd 0:0:23:0: [sdy] Mode Sense: db 00 10 08 [ 21.106951] sd 0:0:20:0: [sdw] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.107074] sd 0:0:27:0: [sdac] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.107111] sd 0:0:24:0: [sdz] Write Protect is off [ 21.107112] sd 0:0:24:0: [sdz] Mode Sense: db 00 10 08 [ 21.107196] sd 0:0:22:0: [sdx] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.107406] sd 0:0:28:0: [sdad] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.107427] sd 0:0:25:0: [sdaa] Write Protect is off [ 21.107433] sd 0:0:25:0: [sdaa] Mode Sense: db 00 10 08 [ 21.107446] sd 0:0:23:0: [sdy] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.107738] sd 0:0:24:0: [sdz] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.107740] sd 0:0:29:0: [sdae] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.107759] sd 0:0:26:0: [sdab] Write Protect is off [ 21.107760] sd 0:0:26:0: [sdab] Mode Sense: db 00 10 08 [ 21.108058] sd 0:0:25:0: [sdaa] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.108146] sd 0:0:30:0: [sdaf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.108180] sd 0:0:27:0: [sdac] Write Protect is off [ 21.108181] sd 0:0:27:0: [sdac] Mode Sense: db 00 10 08 [ 21.108221] sd 0:0:0:0: [sdc] Attached SCSI disk [ 21.108386] sd 0:0:1:0: [sdd] Attached SCSI disk [ 21.108387] sd 0:0:26:0: [sdab] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.108458] sd 0:0:31:0: [sdag] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.108506] sd 0:0:28:0: [sdad] Write Protect is off [ 21.108508] sd 0:0:28:0: [sdad] Mode Sense: db 00 10 08 [ 21.108606] sd 0:0:2:0: [sde] Attached SCSI disk [ 21.108713] sd 0:0:32:0: [sdah] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.108781] sd 0:0:27:0: [sdac] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.108875] sd 0:0:29:0: [sdae] Write Protect is off [ 21.108876] sd 0:0:29:0: [sdae] Mode Sense: db 00 10 08 [ 21.108995] sd 0:0:33:0: [sdai] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.109069] sd 0:0:3:0: [sdf] Attached SCSI disk [ 21.109137] sd 0:0:28:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.109262] sd 0:0:4:0: [sdg] Attached SCSI disk [ 21.109289] sd 0:0:30:0: [sdaf] Write Protect is off [ 21.109290] sd 0:0:30:0: [sdaf] Mode Sense: db 00 10 08 [ 21.109294] sd 0:0:34:0: [sdaj] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.109406] sd 0:0:5:0: [sdh] Attached SCSI disk [ 21.109535] sd 0:0:29:0: [sdae] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.109573] sd 0:0:31:0: [sdag] Write Protect is off [ 21.109575] sd 0:0:31:0: [sdag] Mode Sense: db 00 10 08 [ 21.109626] sd 0:0:6:0: [sdi] Attached SCSI disk [ 21.109640] sd 0:0:35:0: [sdak] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.109795] sd 0:0:32:0: [sdah] Write Protect is off [ 21.109796] sd 0:0:32:0: [sdah] Mode Sense: db 00 10 08 [ 21.109854] sd 0:0:7:0: [sdj] Attached SCSI disk [ 21.109894] sd 0:0:30:0: [sdaf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.109989] sd 0:0:36:0: [sdal] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.110103] sd 0:0:33:0: [sdai] Write Protect is off [ 21.110104] sd 0:0:33:0: [sdai] Mode Sense: db 00 10 08 [ 21.110146] sd 0:0:31:0: [sdag] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.110279] sd 0:0:37:0: [sdam] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.110359] sd 0:0:8:0: [sdk] Attached SCSI disk [ 21.110366] sd 0:0:9:0: [sdl] Attached SCSI disk [ 21.110390] sd 0:0:10:0: [sdm] Attached SCSI disk [ 21.110403] sd 0:0:32:0: [sdah] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.110429] sd 0:0:34:0: [sdaj] Write Protect is off [ 21.110430] sd 0:0:34:0: [sdaj] Mode Sense: db 00 10 08 [ 21.110536] sd 0:0:38:0: [sdan] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.110578] sd 0:0:11:0: [sdn] Attached SCSI disk [ 21.110712] sd 0:0:12:0: [sdo] Attached SCSI disk [ 21.110724] sd 0:0:33:0: [sdai] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.110764] sd 0:0:35:0: [sdak] Write Protect is off [ 21.110765] sd 0:0:35:0: [sdak] Mode Sense: db 00 10 08 [ 21.110803] sd 0:0:39:0: [sdao] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.110997] sd 0:0:34:0: [sdaj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.111056] sd 0:0:13:0: [sdp] Attached SCSI disk [ 21.111113] sd 0:0:36:0: [sdal] Write Protect is off [ 21.111114] sd 0:0:36:0: [sdal] Mode Sense: db 00 10 08 [ 21.111197] sd 0:0:40:0: [sdap] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.111204] sd 0:0:14:0: [sdq] Attached SCSI disk [ 21.111354] sd 0:0:37:0: [sdam] Write Protect is off [ 21.111355] sd 0:0:37:0: [sdam] Mode Sense: db 00 10 08 [ 21.111383] sd 0:0:41:0: [sdaq] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.111403] sd 0:0:35:0: [sdak] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.111620] sd 0:0:42:0: [sdar] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.111633] sd 0:0:38:0: [sdan] Write Protect is off [ 21.111635] sd 0:0:38:0: [sdan] Mode Sense: db 00 10 08 [ 21.111678] sd 0:0:36:0: [sdal] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.111730] sd 0:0:15:0: [sdr] Attached SCSI disk [ 21.111862] sd 0:0:16:0: [sds] Attached SCSI disk [ 21.111906] sd 0:0:39:0: [sdao] Write Protect is off [ 21.111908] sd 0:0:39:0: [sdao] Mode Sense: db 00 10 08 [ 21.111910] sd 0:0:43:0: [sdas] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.111954] sd 0:0:37:0: [sdam] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.112193] sd 0:0:44:0: [sdat] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.112228] sd 0:0:38:0: [sdan] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.112275] sd 0:0:40:0: [sdap] Write Protect is off [ 21.112276] sd 0:0:40:0: [sdap] Mode Sense: db 00 10 08 [ 21.112422] sd 0:0:18:0: [sdu] Attached SCSI disk [ 21.112442] sd 0:0:45:0: [sdau] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 21.112448] sd 0:0:41:0: [sdaq] Write Protect is off [ 21.112450] sd 0:0:41:0: [sdaq] Mode Sense: db 00 10 08 [ 21.112502] sd 0:0:39:0: [sdao] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.112699] sd 0:0:19:0: [sdv] Attached SCSI disk [ 21.112702] sd 0:0:42:0: [sdar] Write Protect is off [ 21.112703] sd 0:0:42:0: [sdar] Mode Sense: db 00 10 08 [ 21.112863] sd 0:0:40:0: [sdap] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.113026] sd 0:0:41:0: [sdaq] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.113027] sd 0:0:43:0: [sdas] Write Protect is off [ 21.113028] sd 0:0:43:0: [sdas] Mode Sense: db 00 10 08 [ 21.113141] sd 0:0:20:0: [sdw] Attached SCSI disk [ 21.113280] sd 0:0:44:0: [sdat] Write Protect is off [ 21.113281] sd 0:0:44:0: [sdat] Mode Sense: db 00 10 08 [ 21.113314] sd 0:0:42:0: [sdar] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.113423] sd 0:0:22:0: [sdx] Attached SCSI disk [ 21.113510] sd 0:0:45:0: [sdau] Write Protect is off [ 21.113511] sd 0:0:45:0: [sdau] Mode Sense: db 00 10 08 [ 21.113567] sd 0:0:23:0: [sdy] Attached SCSI disk [ 21.113594] sd 0:0:43:0: [sdas] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.113843] sd 0:0:44:0: [sdat] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.113944] sd 0:0:24:0: [sdz] Attached SCSI disk [ 21.114061] sd 0:0:45:0: [sdau] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 21.114210] sd 0:0:25:0: [sdaa] Attached SCSI disk [ 21.114520] sd 0:0:26:0: [sdab] Attached SCSI disk [ 21.114931] sd 0:0:27:0: [sdac] Attached SCSI disk [ 21.115333] sd 0:0:28:0: [sdad] Attached SCSI disk [ 21.115485] sd 0:0:29:0: [sdae] Attached SCSI disk [ 21.115994] sd 0:0:30:0: [sdaf] Attached SCSI disk [ 21.116105] sd 0:0:31:0: [sdag] Attached SCSI disk [ 21.116346] sd 0:0:32:0: [sdah] Attached SCSI disk [ 21.116680] sd 0:0:33:0: [sdai] Attached SCSI disk [ 21.117023] sd 0:0:34:0: [sdaj] Attached SCSI disk [ 21.117343] sd 0:0:35:0: [sdak] Attached SCSI disk [ 21.117685] sd 0:0:36:0: [sdal] Attached SCSI disk [ 21.117952] sd 0:0:37:0: [sdam] Attached SCSI disk [ 21.118294] sd 0:0:38:0: [sdan] Attached SCSI disk [ 21.118486] sd 0:0:39:0: [sdao] Attached SCSI disk [ 21.118939] sd 0:0:41:0: [sdaq] Attached SCSI disk [ 21.119036] sd 0:0:40:0: [sdap] Attached SCSI disk [ 21.119204] sd 0:0:42:0: [sdar] Attached SCSI disk [ 21.119318] sd 0:0:43:0: [sdas] Attached SCSI disk [ 21.119594] sd 0:0:44:0: [sdat] Attached SCSI disk [ 21.119854] sd 0:0:45:0: [sdau] Attached SCSI disk [ 21.128842] sd 0:0:17:0: [sdt] Attached SCSI disk [ 22.586169] scsi 8:0:43:0: SSP: handle(0x0036), sas_addr(0x5000c5008429ac96), phy(33), device_name(0x5000c5008429ac94) [ 22.599000] scsi 8:0:43:0: SSP: enclosure_logical_id(0x500304800040533f), slot(21) [ 22.608545] scsi 8:0:43:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 22.634892] scsi 8:0:44:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 22.645143] scsi 8:0:44:0: SSP: handle(0x0037), sas_addr(0x5000c5008429f766), phy(34), device_name(0x5000c5008429f764) [ 22.658128] scsi 8:0:44:0: SSP: enclosure_logical_id(0x500304800040533f), slot(22) [ 22.667638] scsi 8:0:44:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 22.693850] scsi 8:0:45:0: Direct-Access SEAGATE ST1000NM0023 0004 PQ: 0 ANSI: 6 [ 22.704221] scsi 8:0:45:0: SSP: handle(0x0038), sas_addr(0x5000c500842a5f16), phy(35), device_name(0x5000c500842a5f14) [ 22.717379] scsi 8:0:45:0: SSP: enclosure_logical_id(0x500304800040533f), slot(23) [ 22.727020] scsi 8:0:45:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) [ 22.751061] scsi 8:0:46:0: Enclosure LSI SAS2X36 0e0b PQ: 0 ANSI: 5 [ 22.761543] scsi 8:0:46:0: set ignore_delay_remove for handle(0x0039) [ 22.769913] scsi 8:0:46:0: SES: handle(0x0039), sas_addr(0x50030480004053bd), phy(36), device_name(0x0000000000000000) [ 22.783139] scsi 8:0:46:0: SES: enclosure_logical_id(0x500304800040533f), slot(24) [ 22.792887] scsi 8:0:46:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(6), cmd_que(1) [ 22.819512] sd 8:0:0:0: [sdav] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.819920] sd 8:0:1:0: [sdaw] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.820344] sd 8:0:2:0: [sdax] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.820418] sd 8:0:3:0: [sday] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.820955] sd 8:0:4:0: [sdaz] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.821201] sd 8:0:5:0: [sdba] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.821204] sd 8:0:1:0: [sdaw] Write Protect is off [ 22.821207] sd 8:0:1:0: [sdaw] Mode Sense: db 00 10 08 [ 22.821484] sd 8:0:6:0: [sdbb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.821585] sd 8:0:3:0: [sday] Write Protect is off [ 22.821588] sd 8:0:3:0: [sday] Mode Sense: db 00 10 08 [ 22.821638] sd 8:0:2:0: [sdax] Write Protect is off [ 22.821640] sd 8:0:2:0: [sdax] Mode Sense: db 00 10 08 [ 22.821693] sd 8:0:7:0: [sdbc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.821820] sd 8:0:1:0: [sdaw] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.821969] sd 8:0:8:0: [sdbd] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.822116] sd 8:0:9:0: [sdbe] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.822174] sd 8:0:4:0: [sdaz] Write Protect is off [ 22.822199] sd 8:0:4:0: [sdaz] Mode Sense: db 00 10 08 [ 22.822262] sd 8:0:2:0: [sdax] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.822382] sd 8:0:3:0: [sday] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.822446] sd 8:0:10:0: [sdbf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.822477] sd 8:0:5:0: [sdba] Write Protect is off [ 22.822479] sd 8:0:5:0: [sdba] Mode Sense: db 00 10 08 [ 22.822673] sd 8:0:11:0: [sdbg] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.822701] sd 8:0:6:0: [sdbb] Write Protect is off [ 22.822703] sd 8:0:6:0: [sdbb] Mode Sense: db 00 10 08 [ 22.822803] sd 8:0:4:0: [sdaz] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.822967] sd 8:0:7:0: [sdbc] Write Protect is off [ 22.822969] sd 8:0:7:0: [sdbc] Mode Sense: db 00 10 08 [ 22.823097] sd 8:0:5:0: [sdba] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.823152] sd 8:0:13:0: [sdbi] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.823208] sd 8:0:8:0: [sdbd] Write Protect is off [ 22.823211] sd 8:0:8:0: [sdbd] Mode Sense: db 00 10 08 [ 22.823329] sd 8:0:6:0: [sdbb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.823391] sd 8:0:9:0: [sdbe] Write Protect is off [ 22.823395] sd 8:0:9:0: [sdbe] Mode Sense: db 00 10 08 [ 22.823428] sd 8:0:14:0: [sdbj] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.823589] sd 8:0:7:0: [sdbc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.823595] sd 8:0:10:0: [sdbf] Write Protect is off [ 22.823597] sd 8:0:10:0: [sdbf] Mode Sense: db 00 10 08 [ 22.823766] sd 8:0:15:0: [sdbk] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.823835] sd 8:0:11:0: [sdbg] Write Protect is off [ 22.823837] sd 8:0:8:0: [sdbd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.823839] sd 8:0:11:0: [sdbg] Mode Sense: db 00 10 08 [ 22.823941] sd 8:0:16:0: [sdbl] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.824007] sd 8:0:9:0: [sdbe] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.824196] sd 8:0:10:0: [sdbf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.824267] sd 8:0:17:0: [sdbm] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.824348] sd 8:0:13:0: [sdbi] Write Protect is off [ 22.824351] sd 8:0:13:0: [sdbi] Mode Sense: db 00 10 08 [ 22.824449] sd 8:0:11:0: [sdbg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.824473] sd 8:0:18:0: [sdbn] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.824657] sd 8:0:14:0: [sdbj] Write Protect is off [ 22.824660] sd 8:0:14:0: [sdbj] Mode Sense: db 00 10 08 [ 22.824757] sd 8:0:19:0: [sdbo] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.824927] sd 8:0:15:0: [sdbk] Write Protect is off [ 22.824929] sd 8:0:15:0: [sdbk] Mode Sense: db 00 10 08 [ 22.824995] sd 8:0:13:0: [sdbi] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.825086] sd 8:0:16:0: [sdbl] Write Protect is off [ 22.825088] sd 8:0:16:0: [sdbl] Mode Sense: db 00 10 08 [ 22.825263] sd 8:0:14:0: [sdbj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.825442] sd 8:0:17:0: [sdbm] Write Protect is off [ 22.825445] sd 8:0:17:0: [sdbm] Mode Sense: db 00 10 08 [ 22.825540] sd 8:0:15:0: [sdbk] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.825639] sd 8:0:18:0: [sdbn] Write Protect is off [ 22.825642] sd 8:0:18:0: [sdbn] Mode Sense: db 00 10 08 [ 22.825940] sd 8:0:19:0: [sdbo] Write Protect is off [ 22.825942] sd 8:0:19:0: [sdbo] Mode Sense: db 00 10 08 [ 22.826012] sd 8:0:16:0: [sdbl] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.826064] sd 8:0:17:0: [sdbm] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.826213] sd 8:0:22:0: [sdbq] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.826266] sd 8:0:18:0: [sdbn] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.827851] sd 8:0:22:0: [sdbq] Write Protect is off [ 22.827854] sd 8:0:22:0: [sdbq] Mode Sense: db 00 10 08 [ 22.828160] sd 8:0:19:0: [sdbo] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.828361] sd 8:0:1:0: [sdaw] Attached SCSI disk [ 22.828444] sd 8:0:22:0: [sdbq] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.828474] sd 8:0:23:0: [sdbr] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.828643] sd 8:0:2:0: [sdax] Attached SCSI disk [ 22.828707] sd 8:0:24:0: [sdbs] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.828827] sd 8:0:25:0: [sdbt] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.828940] sd 8:0:20:0: [sdbp] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.829043] sd 8:0:4:0: [sdaz] Attached SCSI disk [ 22.829068] sd 8:0:26:0: [sdbu] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.829160] sd 8:0:27:0: [sdbv] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.829388] sd 8:0:28:0: [sdbw] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.829581] sd 8:0:29:0: [sdbx] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.829688] sd 8:0:23:0: [sdbr] Write Protect is off [ 22.829690] sd 8:0:23:0: [sdbr] Mode Sense: db 00 10 08 [ 22.829780] sd 8:0:6:0: [sdbb] Attached SCSI disk [ 22.829843] sd 8:0:30:0: [sdby] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.829872] sd 8:0:24:0: [sdbs] Write Protect is off [ 22.829874] sd 8:0:24:0: [sdbs] Mode Sense: db 00 10 08 [ 22.829989] sd 8:0:3:0: [sday] Attached SCSI disk [ 22.830077] sd 8:0:25:0: [sdbt] Write Protect is off [ 22.830080] sd 8:0:25:0: [sdbt] Mode Sense: db 00 10 08 [ 22.830105] sd 8:0:20:0: [sdbp] Write Protect is off [ 22.830107] sd 8:0:20:0: [sdbp] Mode Sense: db 00 10 08 [ 22.830125] sd 8:0:31:0: [sdbz] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.830245] sd 8:0:26:0: [sdbu] Write Protect is off [ 22.830248] sd 8:0:26:0: [sdbu] Mode Sense: db 00 10 08 [ 22.830290] sd 8:0:32:0: [sdca] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.830293] sd 8:0:23:0: [sdbr] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.830308] sd 8:0:8:0: [sdbd] Attached SCSI disk [ 22.830448] sd 8:0:27:0: [sdbv] Write Protect is off [ 22.830450] sd 8:0:27:0: [sdbv] Mode Sense: db 00 10 08 [ 22.830571] sd 8:0:28:0: [sdbw] Write Protect is off [ 22.830573] sd 8:0:28:0: [sdbw] Mode Sense: db 00 10 08 [ 22.830676] sd 8:0:25:0: [sdbt] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.830709] sd 8:0:29:0: [sdbx] Write Protect is off [ 22.830712] sd 8:0:29:0: [sdbx] Mode Sense: db 00 10 08 [ 22.830722] sd 8:0:20:0: [sdbp] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.830802] sd 8:0:33:0: [sdcb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.830845] sd 8:0:26:0: [sdbu] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.830959] sd 8:0:5:0: [sdba] Attached SCSI disk [ 22.831000] sd 8:0:30:0: [sdby] Write Protect is off [ 22.831001] sd 8:0:30:0: [sdby] Mode Sense: db 00 10 08 [ 22.831011] sd 8:0:7:0: [sdbc] Attached SCSI disk [ 22.831060] sd 8:0:27:0: [sdbv] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.831227] sd 8:0:28:0: [sdbw] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.831301] sd 8:0:29:0: [sdbx] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.831307] sd 8:0:31:0: [sdbz] Write Protect is off [ 22.831308] sd 8:0:31:0: [sdbz] Mode Sense: db 00 10 08 [ 22.831347] sd 8:0:10:0: [sdbf] Attached SCSI disk [ 22.831356] sd 8:0:24:0: [sdbs] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.831437] sd 8:0:32:0: [sdca] Write Protect is off [ 22.831439] sd 8:0:32:0: [sdca] Mode Sense: db 00 10 08 [ 22.831525] sd 8:0:34:0: [sdcc] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.831587] sd 8:0:9:0: [sdbe] Attached SCSI disk [ 22.831638] sd 8:0:30:0: [sdby] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.831875] sd 8:0:14:0: [sdbj] Attached SCSI disk [ 22.831932] sd 8:0:31:0: [sdbz] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.831959] sd 8:0:13:0: [sdbi] Attached SCSI disk [ 22.832018] sd 8:0:35:0: [sdcd] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.832068] sd 8:0:32:0: [sdca] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.832069] sd 8:0:33:0: [sdcb] Write Protect is off [ 22.832071] sd 8:0:33:0: [sdcb] Mode Sense: db 00 10 08 [ 22.832189] sd 8:0:11:0: [sdbg] Attached SCSI disk [ 22.832290] sd 8:0:36:0: [sdce] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.832643] sd 8:0:37:0: [sdcf] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.832690] sd 8:0:33:0: [sdcb] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.832705] sd 8:0:34:0: [sdcc] Write Protect is off [ 22.832708] sd 8:0:34:0: [sdcc] Mode Sense: db 00 10 08 [ 22.832727] sd 8:0:17:0: [sdbm] Attached SCSI disk [ 22.832850] sd 8:0:15:0: [sdbk] Attached SCSI disk [ 22.833161] sd 8:0:38:0: [sdcg] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.833168] sd 8:0:35:0: [sdcd] Write Protect is off [ 22.833169] sd 8:0:35:0: [sdcd] Mode Sense: db 00 10 08 [ 22.833359] sd 8:0:34:0: [sdcc] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.833434] sd 8:0:36:0: [sdce] Write Protect is off [ 22.833436] sd 8:0:36:0: [sdce] Mode Sense: db 00 10 08 [ 22.833470] sd 8:0:18:0: [sdbn] Attached SCSI disk [ 22.833595] sd 8:0:16:0: [sdbl] Attached SCSI disk [ 22.833625] sd 8:0:39:0: [sdch] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.833789] sd 8:0:37:0: [sdcf] Write Protect is off [ 22.833790] sd 8:0:37:0: [sdcf] Mode Sense: db 00 10 08 [ 22.833847] sd 8:0:35:0: [sdcd] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.834054] sd 8:0:36:0: [sdce] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.834165] sd 8:0:12:0: [sdbh] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.834260] sd 8:0:40:0: [sdci] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.834356] sd 8:0:38:0: [sdcg] Write Protect is off [ 22.834358] sd 8:0:38:0: [sdcg] Mode Sense: db 00 10 08 [ 22.834381] sd 8:0:19:0: [sdbo] Attached SCSI disk [ 22.834439] sd 8:0:37:0: [sdcf] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.834542] sd 8:0:41:0: [sdcj] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.834784] sd 8:0:39:0: [sdch] Write Protect is off [ 22.834785] sd 8:0:39:0: [sdch] Mode Sense: db 00 10 08 [ 22.834922] sd 8:0:22:0: [sdbq] Attached SCSI disk [ 22.834965] sd 8:0:42:0: [sdck] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.834967] sd 8:0:38:0: [sdcg] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.835299] sd 8:0:12:0: [sdbh] Write Protect is off [ 22.835300] sd 8:0:12:0: [sdbh] Mode Sense: db 00 10 08 [ 22.835325] sd 8:0:43:0: [sdcl] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.835424] sd 8:0:39:0: [sdch] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.835514] sd 8:0:44:0: [sdcm] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.835542] sd 8:0:40:0: [sdci] Write Protect is off [ 22.835543] sd 8:0:40:0: [sdci] Mode Sense: db 00 10 08 [ 22.835623] sd 8:0:41:0: [sdcj] Write Protect is off [ 22.835625] sd 8:0:41:0: [sdcj] Mode Sense: db 00 10 08 [ 22.835965] sd 8:0:12:0: [sdbh] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.836038] sd 8:0:42:0: [sdck] Write Protect is off [ 22.836039] sd 8:0:42:0: [sdck] Mode Sense: db 00 10 08 [ 22.836054] sd 8:0:45:0: [sdcn] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB) [ 22.836179] sd 8:0:41:0: [sdcj] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.836404] sd 8:0:43:0: [sdcl] Write Protect is off [ 22.836405] sd 8:0:43:0: [sdcl] Mode Sense: db 00 10 08 [ 22.836530] sd 8:0:40:0: [sdci] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.836594] sd 8:0:44:0: [sdcm] Write Protect is off [ 22.836596] sd 8:0:44:0: [sdcm] Mode Sense: db 00 10 08 [ 22.836612] sd 8:0:42:0: [sdck] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.836710] sd 8:0:23:0: [sdbr] Attached SCSI disk [ 22.836961] sd 8:0:43:0: [sdcl] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.837063] sd 8:0:20:0: [sdbp] Attached SCSI disk [ 22.837083] sd 8:0:25:0: [sdbt] Attached SCSI disk [ 22.837093] sd 8:0:26:0: [sdbu] Attached SCSI disk [ 22.837140] sd 8:0:45:0: [sdcn] Write Protect is off [ 22.837142] sd 8:0:45:0: [sdcn] Mode Sense: db 00 10 08 [ 22.837153] sd 8:0:44:0: [sdcm] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.837339] sd 8:0:28:0: [sdbw] Attached SCSI disk [ 22.837461] sd 8:0:27:0: [sdbv] Attached SCSI disk [ 22.837559] sd 8:0:24:0: [sdbs] Attached SCSI disk [ 22.837626] sd 8:0:30:0: [sdby] Attached SCSI disk [ 22.837644] sd 8:0:29:0: [sdbx] Attached SCSI disk [ 22.837722] sd 8:0:45:0: [sdcn] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 22.838425] sd 8:0:31:0: [sdbz] Attached SCSI disk [ 22.838511] sd 8:0:32:0: [sdca] Attached SCSI disk [ 22.839061] sd 8:0:33:0: [sdcb] Attached SCSI disk [ 22.839750] sd 8:0:34:0: [sdcc] Attached SCSI disk [ 22.840103] sd 8:0:35:0: [sdcd] Attached SCSI disk [ 22.840437] sd 8:0:36:0: [sdce] Attached SCSI disk [ 22.840567] sd 8:0:37:0: [sdcf] Attached SCSI disk [ 22.841297] sd 8:0:38:0: [sdcg] Attached SCSI disk [ 22.841777] sd 8:0:39:0: [sdch] Attached SCSI disk [ 22.842042] sd 8:0:12:0: [sdbh] Attached SCSI disk [ 22.842115] sd 8:0:41:0: [sdcj] Attached SCSI disk [ 22.842455] sd 8:0:42:0: [sdck] Attached SCSI disk [ 22.842578] sd 8:0:40:0: [sdci] Attached SCSI disk [ 22.843052] sd 8:0:44:0: [sdcm] Attached SCSI disk [ 22.843167] sd 8:0:43:0: [sdcl] Attached SCSI disk [ 22.843694] sd 8:0:45:0: [sdcn] Attached SCSI disk [ 24.247198] sd 8:0:0:0: [sdav] Write Protect is off [ 24.253123] sd 8:0:0:0: [sdav] Mode Sense: db 00 10 08 [ 24.253751] sd 8:0:0:0: [sdav] Write cache: enabled, read cache: enabled, supports DPO and FUA [ 24.270721] sd 8:0:0:0: [sdav] Attached SCSI disk [ 27.618000] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem [ 27.648154] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) [ 28.133829] systemd-journald[281]: Received SIGTERM from PID 1 (systemd). [ 28.730441] SELinux: Disabled at runtime. [ 28.736317] SELinux: Unregistering netfilter hooks [ 28.777469] type=1404 audit(1513210148.740:2): selinux=0 auid=4294967295 ses=4294967295 [ 29.025596] ip_tables: (C) 2000-2006 Netfilter Core Team [ 29.033197] systemd[1]: Inserted module 'ip_tables' [ 30.681154] RPC: Registered named UNIX socket transport module. [ 30.689188] RPC: Registered udp transport module. [ 30.696048] RPC: Registered tcp transport module. [ 30.702501] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 30.934190] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 31.485126] EXT4-fs (sda1): re-mounted. Opts: (null) [ 31.552693] systemd-journald[1003]: Received request to flush runtime journal from PID 1 [ 31.612999] device-mapper: uevent: version 1.0.3 [ 31.613160] device-mapper: ioctl: 4.35.0-ioctl (2016-06-23) initialised: dm-devel@redhat.com [ 31.675713] wmi: Mapper loaded [ 31.765182] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 31.848182] ipmi message handler version 39.2 [ 31.866170] ioatdma: Intel(R) QuickData Technology Driver 4.00 [ 31.873863] ipmi device interface [ 31.878957] ioatdma 0000:00:04.0: irq 182 for MSI/MSI-X [ 31.879430] igb 0000:02:00.0: DCA enabled [ 31.884700] igb 0000:02:00.1: DCA enabled [ 31.890200] ioatdma 0000:00:04.1: irq 184 for MSI/MSI-X [ 31.890626] igb 0000:02:00.2: DCA enabled [ 31.895855] igb 0000:02:00.3: DCA enabled [ 31.901267] ioatdma 0000:00:04.2: irq 185 for MSI/MSI-X [ 31.901936] ioatdma 0000:00:04.3: irq 186 for MSI/MSI-X [ 31.902504] ioatdma 0000:00:04.4: irq 187 for MSI/MSI-X [ 31.903087] ioatdma 0000:00:04.5: irq 188 for MSI/MSI-X [ 31.903786] ioatdma 0000:00:04.6: irq 189 for MSI/MSI-X [ 31.904348] ioatdma 0000:00:04.7: irq 190 for MSI/MSI-X [ 31.905035] ioatdma 0000:80:04.0: irq 192 for MSI/MSI-X [ 31.905707] ioatdma 0000:80:04.1: irq 194 for MSI/MSI-X [ 31.906260] ioatdma 0000:80:04.2: irq 195 for MSI/MSI-X [ 31.906820] ioatdma 0000:80:04.3: irq 196 for MSI/MSI-X [ 31.907392] ioatdma 0000:80:04.4: irq 197 for MSI/MSI-X [ 31.907962] ioatdma 0000:80:04.5: irq 198 for MSI/MSI-X [ 31.908484] ioatdma 0000:80:04.6: irq 199 for MSI/MSI-X [ 31.909047] ioatdma 0000:80:04.7: irq 200 for MSI/MSI-X [ 32.001675] ACPI Warning: SystemIO range 0x0000000000004000-0x000000000000401f conflicts with OpRegion 0x0000000000004000-0x000000000000400f (\_SB_.PCI0.SBUS.SMBI) (20130517/utaddress-254) [ 32.022068] ACPI: If an ACPI driver is available for this device, you should use it instead of the native driver [ 32.056264] i801_smbus 0000:07:00.3: Enabling SMBus device [ 32.063231] i801_smbus 0000:07:00.3: SMBus using PCI interrupt [ 32.129908] mei_me 0000:00:16.0: Device doesn't have valid ME Interface [ 32.216039] IPMI System Interface driver. [ 32.225506] sd 7:0:0:0: Attached scsi generic sg0 type 0 [ 32.225573] sd 7:0:1:0: Attached scsi generic sg1 type 0 [ 32.225635] sd 0:0:0:0: Attached scsi generic sg2 type 0 [ 32.225697] sd 0:0:1:0: Attached scsi generic sg3 type 0 [ 32.225756] sd 0:0:2:0: Attached scsi generic sg4 type 0 [ 32.225814] sd 0:0:3:0: Attached scsi generic sg5 type 0 [ 32.225869] sd 0:0:4:0: Attached scsi generic sg6 type 0 [ 32.225932] sd 0:0:5:0: Attached scsi generic sg7 type 0 [ 32.225987] sd 0:0:6:0: Attached scsi generic sg8 type 0 [ 32.226043] sd 0:0:7:0: Attached scsi generic sg9 type 0 [ 32.226099] sd 0:0:8:0: Attached scsi generic sg10 type 0 [ 32.226155] sd 0:0:9:0: Attached scsi generic sg11 type 0 [ 32.226211] sd 0:0:10:0: Attached scsi generic sg12 type 0 [ 32.226268] sd 0:0:11:0: Attached scsi generic sg13 type 0 [ 32.226324] sd 0:0:12:0: Attached scsi generic sg14 type 0 [ 32.226379] sd 0:0:13:0: Attached scsi generic sg15 type 0 [ 32.226435] sd 0:0:14:0: Attached scsi generic sg16 type 0 [ 32.226491] sd 0:0:15:0: Attached scsi generic sg17 type 0 [ 32.226551] sd 0:0:16:0: Attached scsi generic sg18 type 0 [ 32.226608] sd 0:0:17:0: Attached scsi generic sg19 type 0 [ 32.226664] sd 0:0:18:0: Attached scsi generic sg20 type 0 [ 32.226719] sd 0:0:19:0: Attached scsi generic sg21 type 0 [ 32.226776] sd 0:0:20:0: Attached scsi generic sg22 type 0 [ 32.226831] scsi 0:0:21:0: Attached scsi generic sg23 type 13 [ 32.226887] sd 0:0:22:0: Attached scsi generic sg24 type 0 [ 32.226943] sd 0:0:23:0: Attached scsi generic sg25 type 0 [ 32.226999] sd 0:0:24:0: Attached scsi generic sg26 type 0 [ 32.227055] sd 0:0:25:0: Attached scsi generic sg27 type 0 [ 32.227112] sd 0:0:26:0: Attached scsi generic sg28 type 0 [ 32.227168] sd 0:0:27:0: Attached scsi generic sg29 type 0 [ 32.227224] sd 0:0:28:0: Attached scsi generic sg30 type 0 [ 32.227282] sd 0:0:29:0: Attached scsi generic sg31 type 0 [ 32.227343] sd 0:0:30:0: Attached scsi generic sg32 type 0 [ 32.227398] sd 0:0:31:0: Attached scsi generic sg33 type 0 [ 32.227457] sd 0:0:32:0: Attached scsi generic sg34 type 0 [ 32.227512] sd 0:0:33:0: Attached scsi generic sg35 type 0 [ 32.227577] sd 0:0:34:0: Attached scsi generic sg36 type 0 [ 32.227621] ipmi_si: probing via SMBIOS [ 32.227625] ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0 [ 32.227627] ipmi_si: Adding SMBIOS-specified kcs state machine [ 32.227633] sd 0:0:35:0: Attached scsi generic sg37 type 0 [ 32.227635] ipmi_si: probing via SPMI [ 32.227637] ipmi_si: SPMI: io 0xca2 regsize 1 spacing 1 irq 0 [ 32.227638] ipmi_si: Adding SPMI-specified kcs state machine duplicate interface [ 32.227639] ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca2, slave address 0x20, irq 0 [ 32.227694] sd 0:0:36:0: Attached scsi generic sg38 type 0 [ 32.227751] sd 0:0:37:0: Attached scsi generic sg39 type 0 [ 32.227808] sd 0:0:38:0: Attached scsi generic sg40 type 0 [ 32.227867] sd 0:0:39:0: Attached scsi generic sg41 type 0 [ 32.227922] sd 0:0:40:0: Attached scsi generic sg42 type 0 [ 32.227979] sd 0:0:41:0: Attached scsi generic sg43 type 0 [ 32.228035] sd 0:0:42:0: Attached scsi generic sg44 type 0 [ 32.228097] sd 0:0:43:0: Attached scsi generic sg45 type 0 [ 32.228152] sd 0:0:44:0: Attached scsi generic sg46 type 0 [ 32.228209] sd 0:0:45:0: Attached scsi generic sg47 type 0 [ 32.228264] scsi 0:0:46:0: Attached scsi generic sg48 type 13 [ 32.228324] sd 8:0:0:0: Attached scsi generic sg49 type 0 [ 32.228379] sd 8:0:1:0: Attached scsi generic sg50 type 0 [ 32.228438] sd 8:0:2:0: Attached scsi generic sg51 type 0 [ 32.228496] sd 8:0:3:0: Attached scsi generic sg52 type 0 [ 32.228566] sd 8:0:4:0: Attached scsi generic sg53 type 0 [ 32.228623] sd 8:0:5:0: Attached scsi generic sg54 type 0 [ 32.228685] sd 8:0:6:0: Attached scsi generic sg55 type 0 [ 32.228741] sd 8:0:7:0: Attached scsi generic sg56 type 0 [ 32.228803] sd 8:0:8:0: Attached scsi generic sg57 type 0 [ 32.228862] sd 8:0:9:0: Attached scsi generic sg58 type 0 [ 32.228917] sd 8:0:10:0: Attached scsi generic sg59 type 0 [ 32.228977] sd 8:0:11:0: Attached scsi generic sg60 type 0 [ 32.229033] sd 8:0:12:0: Attached scsi generic sg61 type 0 [ 32.229092] sd 8:0:13:0: Attached scsi generic sg62 type 0 [ 32.229148] sd 8:0:14:0: Attached scsi generic sg63 type 0 [ 32.229209] sd 8:0:15:0: Attached scsi generic sg64 type 0 [ 32.229265] sd 8:0:16:0: Attached scsi generic sg65 type 0 [ 32.229326] sd 8:0:17:0: Attached scsi generic sg66 type 0 [ 32.229382] sd 8:0:18:0: Attached scsi generic sg67 type 0 [ 32.229441] sd 8:0:19:0: Attached scsi generic sg68 type 0 [ 32.229497] sd 8:0:20:0: Attached scsi generic sg69 type 0 [ 32.229562] scsi 8:0:21:0: Attached scsi generic sg70 type 13 [ 32.229620] sd 8:0:22:0: Attached scsi generic sg71 type 0 [ 32.229679] sd 8:0:23:0: Attached scsi generic sg72 type 0 [ 32.229735] sd 8:0:24:0: Attached scsi generic sg73 type 0 [ 32.229794] sd 8:0:25:0: Attached scsi generic sg74 type 0 [ 32.229852] sd 8:0:26:0: Attached scsi generic sg75 type 0 [ 32.229909] sd 8:0:27:0: Attached scsi generic sg76 type 0 [ 32.229973] sd 8:0:28:0: Attached scsi generic sg77 type 0 [ 32.230030] sd 8:0:29:0: Attached scsi generic sg78 type 0 [ 32.230089] sd 8:0:30:0: Attached scsi generic sg79 type 0 [ 32.230148] sd 8:0:31:0: Attached scsi generic sg80 type 0 [ 32.230207] sd 8:0:32:0: Attached scsi generic sg81 type 0 [ 32.230263] sd 8:0:33:0: Attached scsi generic sg82 type 0 [ 32.230322] sd 8:0:34:0: Attached scsi generic sg83 type 0 [ 32.230381] sd 8:0:35:0: Attached scsi generic sg84 type 0 [ 32.230440] sd 8:0:36:0: Attached scsi generic sg85 type 0 [ 32.230495] sd 8:0:37:0: Attached scsi generic sg86 type 0 [ 32.230560] sd 8:0:38:0: Attached scsi generic sg87 type 0 [ 32.230618] sd 8:0:39:0: Attached scsi generic sg88 type 0 [ 32.230678] sd 8:0:40:0: Attached scsi generic sg89 type 0 [ 32.230736] sd 8:0:41:0: Attached scsi generic sg90 type 0 [ 32.230797] sd 8:0:42:0: Attached scsi generic sg91 type 0 [ 32.230855] sd 8:0:43:0: Attached scsi generic sg92 type 0 [ 32.230915] sd 8:0:44:0: Attached scsi generic sg93 type 0 [ 32.230985] sd 8:0:45:0: Attached scsi generic sg94 type 0 [ 32.231044] scsi 8:0:46:0: Attached scsi generic sg95 type 13 [ 32.276923] ipmi_si ipmi_si.0: Found new BMC (man_id: 0x000157, prod_id: 0x0049, dev_id: 0x21) [ 32.277218] ipmi_si ipmi_si.0: IPMI kcs interface initialized [ 32.380081] IPMI SSIF Interface driver [ 33.305718] device-mapper: multipath round-robin: version 1.2.0 loaded [ 33.409108] input: PC Speaker as /devices/platform/pcspkr/input/input3 [ 33.492963] iTCO_vendor_support: vendor-support=0 [ 33.501210] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11 [ 33.507862] iTCO_wdt: unable to reset NO_REBOOT flag, device disabled by hardware/BIOS [ 33.552477] ses 0:0:21:0: Attached Enclosure device [ 33.558491] ses 0:0:46:0: Attached Enclosure device [ 33.564320] ses 8:0:21:0: Attached Enclosure device [ 33.570138] ses 8:0:46:0: Attached Enclosure device [ 34.041649] device-mapper: table: 253:91: multipath: error getting device [ 34.050412] device-mapper: ioctl: error adding target to table [ 34.165253] AES CTR mode by8 optimization enabled [ 34.176736] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni) [ 34.492725] alg: No test for crc32 (crc32-pclmul) [ 34.576019] Adding 16319484k swap on /dev/sda2. Priority:-1 extents:1 across:16319484k FS [ 34.741288] type=1305 audit(1513210154.703:3): audit_pid=2539 old=0 auid=4294967295 ses=4294967295 res=1 [ 35.062586] kvm: disabled by bios [ 35.097205] kvm: disabled by bios [ 35.132624] intel_rapl: Found RAPL domain package [ 35.132627] intel_rapl: Found RAPL domain core [ 35.132638] intel_rapl: Found RAPL domain dram [ 35.132653] intel_rapl: Found RAPL domain package [ 35.132654] intel_rapl: Found RAPL domain core [ 35.132667] intel_rapl: Found RAPL domain dram [ 35.328169] SPL: Loaded module v0.7.3-1 [ 35.582922] znvpair: module license 'CDDL' taints kernel. [ 35.582923] Disabling lock debugging due to kernel taint [ 35.731766] EDAC MC: Ver: 3.0.0 [ 35.777414] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 35.777438] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 35.777447] EDAC sbridge: Seeking for: PCI ID 8086:3ca0 [ 35.777452] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 35.777458] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 35.777462] EDAC sbridge: Seeking for: PCI ID 8086:3ca8 [ 35.777464] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 35.777470] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 35.777474] EDAC sbridge: Seeking for: PCI ID 8086:3c71 [ 35.777475] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 35.777481] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 35.777485] EDAC sbridge: Seeking for: PCI ID 8086:3caa [ 35.777487] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 35.777493] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 35.777497] EDAC sbridge: Seeking for: PCI ID 8086:3cab [ 35.777498] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 35.777504] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 35.777508] EDAC sbridge: Seeking for: PCI ID 8086:3cac [ 35.777510] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 35.777516] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 35.777520] EDAC sbridge: Seeking for: PCI ID 8086:3cad [ 35.777521] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 35.777528] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 35.777532] EDAC sbridge: Seeking for: PCI ID 8086:3cb8 [ 35.777533] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 35.777538] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 35.777542] EDAC sbridge: Seeking for: PCI ID 8086:3cf4 [ 35.777544] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 35.777551] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 35.777555] EDAC sbridge: Seeking for: PCI ID 8086:3cf6 [ 35.777557] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 35.777572] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 35.777576] EDAC sbridge: Seeking for: PCI ID 8086:3cf5 [ 35.777745] EDAC MC0: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#0': DEV 0000:7f:0e.0 [ 35.777863] EDAC MC1: Giving out device to 'sbridge_edac.c' 'Sandy Bridge Socket#1': DEV 0000:ff:0e.0 [ 35.777864] EDAC sbridge: Ver: 1.1.1 [ 37.145509] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 37.234953] igb 0000:02:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 37.242472] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready [ 37.324864] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready [ 37.336108] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready [ 37.454955] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready [ 37.466097] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready [ 37.585355] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready [ 38.190824] ZFS: Loaded module v0.7.3-1, ZFS pool version 5000, ZFS filesystem version 5 [ 38.894008] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.2-1.0.0 [ 39.286278] card: mlx4_0, QP: 0x220, inline size: 120 [ 39.310683] card: mlx4_0, QP: 0x300, inline size: 120 [ 39.340232] ib0: enabling connected mode will cause multicast packet drops [ 39.340550] ib0: mtu > 4092 will cause multicast packet drops. [ 39.835777] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 39.861476] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 39.872534] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [ 39.896543] IPv6: ADDRCONF(NETDEV_UP): ib1: link is not ready [ 46.407655] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 46.465699] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 46.475388] IPv6: ADDRCONF(NETDEV_UP): ib0: link is not ready [ 46.721391] FS-Cache: Loaded [ 46.742200] FS-Cache: Netfs 'nfs' registered for caching [ 46.746430] Key type dns_resolver registered [ 46.776604] NFS: Registering the id_resolver key type [ 46.776607] Key type id_resolver registered [ 46.776607] Key type id_legacy registered [ 59.564939] IPv6: ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready [ 224.326383] LNet: HW NUMA nodes: 2, HW CPU cores: 32, npartitions: 2 [ 224.350130] alg: No test for adler32 (adler32-zlib) [ 224.366392] alg: No test for crc32 (crc32-table) [ 225.218334] Lustre: Lustre: Build Version: 2.10.2_RC1_4_g2022d41 [ 225.419922] LNet: Added LNI 192.168.1.107@o2ib [8/256/0/180] [ 226.017862] Lustre: soaked-OST0017: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 231.128683] Lustre: soaked-OST0017: Will be in recovery for at least 2:30, or until 33 clients reconnect [ 231.160082] Lustre: soaked-OST0017: Connection restored to soaked-MDT0001-mdtlov_UUID (at 192.168.1.109@o2ib) [ 231.407375] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 4294898 seconds [ 231.833665] Lustre: soaked-OST0017: Connection restored to 98054527-4337-40ac-baa5-49c44ee20102 (at 192.168.1.135@o2ib) [ 231.869280] Lustre: Skipped 1 previous similar message [ 232.377204] Lustre: soaked-OST0005: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 232.407400] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 4294899 seconds [ 232.511233] Lustre: soaked-OST0005: Will be in recovery for at least 2:30, or until 33 clients reconnect [ 232.842048] Lustre: soaked-OST0017: Connection restored to 786e75d9-48fb-9730-acb4-8151438bd73f (at 192.168.1.120@o2ib) [ 232.877655] Lustre: Skipped 11 previous similar messages [ 234.851497] Lustre: soaked-OST0005: Connection restored to 9d9e7df2-293b-033f-3d78-e6db9c11dd73 (at 192.168.1.117@o2ib) [ 234.887087] Lustre: Skipped 26 previous similar messages [ 236.054596] Lustre: soaked-OST0017: Recovery over after 0:05, of 33 clients 33 recovered and 0 were evicted. [ 236.071683] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9019721 to 0x8c0000400:9031634 [ 236.093836] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10603562 to 0x8c0000401:10603585 [ 236.107418] Lustre: soaked-OST0017: deleting orphan objects from 0x0:24523097 to 0x0:24523126 [ 236.120451] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000402:9306522 to 0x8c0000402:9306546 [ 238.752608] Lustre: soaked-OST000b: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 240.587326] Lustre: soaked-OST000b: Will be in recovery for at least 2:30, or until 33 clients reconnect [ 240.587391] Lustre: soaked-OST0005: Connection restored to 79cd3d2c-8d26-b508-dda7-50bc184269cf (at 192.168.1.125@o2ib) [ 240.587394] Lustre: Skipped 19 previous similar messages [ 245.097482] Lustre: soaked-OST0011: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 [ 245.313705] Lustre: soaked-OST0011: Will be in recovery for at least 2:30, or until 33 clients reconnect [ 248.646022] Lustre: soaked-OST000b: Recovery over after 0:08, of 33 clients 33 recovered and 0 were evicted. [ 248.724952] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10617483 to 0x5c0000402:10617505 [ 248.724954] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000400:9300312 to 0x5c0000400:9300337 [ 248.754325] Lustre: soaked-OST000b: deleting orphan objects from 0x0:24617257 to 0x0:24617285 [ 248.757042] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9031372 to 0x5c0000401:9035505 [ 251.368576] Lustre: soaked-OST0011: Connection restored to 786e75d9-48fb-9730-acb4-8151438bd73f (at 192.168.1.120@o2ib) [ 251.404201] Lustre: Skipped 61 previous similar messages [ 255.541519] Lustre: soaked-OST0011: Recovery over after 0:10, of 33 clients 33 recovered and 0 were evicted. [ 255.569227] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10605866 to 0x740000402:10605889 [ 255.572223] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9270920 to 0x740000401:9270945 [ 255.583249] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9024978 to 0x740000400:9027393 [ 255.585168] Lustre: soaked-OST0011: deleting orphan objects from 0x0:24577445 to 0x0:24577473 [ 262.081297] Lustre: soaked-OST0005: Recovery over after 0:30, of 33 clients 33 recovered and 0 were evicted. [ 262.173491] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9047049 to 0x440000400:9051410 [ 262.192280] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9331657 to 0x440000402:9331682 [ 262.195044] Lustre: soaked-OST0005: deleting orphan objects from 0x0:24582825 to 0x0:24582852 [ 262.292317] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10624970 to 0x440000401:10624993 [ 269.004735] LustreError: 15767:0:(osd_index.c:224:__osd_xattr_load_by_oid()) soaked-OST000b: can't get bonus, rc = -17 [ 286.860376] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) soaked-OST0005: can't get bonus, rc = -17 [ 286.896952] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) Skipped 7 previous similar messages [ 288.619338] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) soaked-OST0005: can't get bonus, rc = -17 [ 288.655896] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) Skipped 39 previous similar messages [ 290.669301] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) soaked-OST0005: can't get bonus, rc = -17 [ 290.705988] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) Skipped 4031 previous similar messages [ 319.051816] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) soaked-OST0005: can't get bonus, rc = -17 [ 319.088217] LustreError: 22360:0:(osd_index.c:224:__osd_xattr_load_by_oid()) Skipped 2521 previous similar messages [ 600.906181] INFO: task ll_ost_io00_000:5763 blocked for more than 120 seconds. [ 600.995365] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.091509] ll_ost_io00_000 D 0000000000000000 0 5763 2 0x00000080 [ 601.178335] ffff88082420b868 0000000000000046 ffff88080c5d4f10 ffff88082420bfd8 [ 601.269260] ffff88082420bfd8 ffff88082420bfd8 ffff88080c5d4f10 ffff8800369f2260 [ 601.360123] ffff8800369f2170 ffff8800369f2198 ffff8800369f2268 0000000000000000 [ 601.447018] Call Trace: [ 601.447022] [ 601.447031] [] schedule+0x29/0x70 [ 601.447041] [ 601.447074] [] cv_wait_common+0x125/0x150 [spl] [ 601.447075] [ 601.447081] [] ? wake_up_atomic_t+0x30/0x30 [ 601.447083] [ 601.447090] [] __cv_wait+0x15/0x20 [spl] [ 601.447091] [ 601.447136] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.447137] [ 601.447169] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.447170] [ 601.447182] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.447183] [ 601.447195] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 601.447195] [ 601.447204] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 601.447205] [ 601.447213] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 601.447214] [ 601.447299] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 601.447300] [ 601.447373] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 601.447374] [ 601.447378] [] ? update_curr+0x104/0x190 [ 601.447379] [ 601.447382] [] ? __enqueue_entity+0x78/0x80 [ 601.447382] [ 601.447385] [] ? enqueue_entity+0x26c/0xb60 [ 601.447386] [ 601.447428] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [ 601.447430] [ 601.447487] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.447488] [ 601.447540] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.447541] [ 601.447590] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.447591] [ 601.447594] [] ? default_wake_function+0x12/0x20 [ 601.447594] [ 601.447598] [] ? __wake_up_common+0x58/0x90 [ 601.447599] [ 601.447648] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.447649] [ 601.447653] [] ? __switch_to+0xd7/0x510 [ 601.447653] [ 601.447655] [] ? __schedule+0x370/0x8b0 [ 601.447656] [ 601.447705] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.447706] [ 601.447708] [] kthread+0xcf/0xe0 [ 601.447709] [ 601.447711] [] ? insert_kthread_work+0x40/0x40 [ 601.447711] [ 601.447714] [] ret_from_fork+0x58/0x90 [ 601.447715] [ 601.447716] [] ? insert_kthread_work+0x40/0x40 [ 601.447719] INFO: task ll_ost_io00_001:5764 blocked for more than 120 seconds. [ 601.447719] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.447720] ll_ost_io00_001 D [ 601.447721] 0000000000000000 [ 601.447722] 0 5764 2 0x00000080 [ 601.447723] ffff88080b0a3868 [ 601.447724] 0000000000000046 [ 601.447724] ffff88080c5d5ee0 [ 601.447725] ffff88080b0a3fd8 [ 601.447725] [ 601.447725] ffff88080b0a3fd8 [ 601.447726] ffff88080b0a3fd8 [ 601.447726] ffff88080c5d5ee0 [ 601.447727] ffff8800369f2260 [ 601.447727] [ 601.447728] ffff8800369f2170 [ 601.447728] ffff8800369f2198 [ 601.447729] ffff8800369f2268 [ 601.447729] 0000000000000000 [ 601.447730] [ 601.447730] Call Trace: [ 601.447731] [ 601.447733] [] schedule+0x29/0x70 [ 601.447736] [ 601.447743] [] cv_wait_common+0x125/0x150 [spl] [ 601.447743] [ 601.447745] [] ? wake_up_atomic_t+0x30/0x30 [ 601.447747] [ 601.447752] [] __cv_wait+0x15/0x20 [spl] [ 601.447753] [ 601.447788] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.447789] [ 601.447821] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.447822] [ 601.447828] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.447829] [ 601.447838] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 601.447838] [ 601.447847] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 601.447847] [ 601.447855] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 601.447856] [ 601.447916] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 601.447917] [ 601.447975] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 601.447981] [ 601.447985] [] ? update_curr+0x104/0x190 [ 601.447985] [ 601.447988] [] ? __enqueue_entity+0x78/0x80 [ 601.447988] [ 601.447991] [] ? enqueue_entity+0x26c/0xb60 [ 601.447992] [ 601.448034] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [ 601.448035] [ 601.448090] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.448091] [ 601.448142] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.448143] [ 601.448192] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.448192] [ 601.448195] [] ? default_wake_function+0x12/0x20 [ 601.448195] [ 601.448197] [] ? __wake_up_common+0x58/0x90 [ 601.448198] [ 601.448248] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.448249] [ 601.448297] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.448298] [ 601.448300] [] kthread+0xcf/0xe0 [ 601.448301] [ 601.448302] [] ? insert_kthread_work+0x40/0x40 [ 601.448303] [ 601.448305] [] ret_from_fork+0x58/0x90 [ 601.448305] [ 601.448307] [] ? insert_kthread_work+0x40/0x40 [ 601.448310] INFO: task ll_ost_io01_000:5766 blocked for more than 120 seconds. [ 601.448311] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.448311] ll_ost_io01_000 D [ 601.448312] 0000000000000000 [ 601.448313] 0 5766 2 0x00000080 [ 601.448314] ffff88080b0b3868 [ 601.448315] 0000000000000046 [ 601.448315] ffff88080b0a8000 [ 601.448316] ffff88080b0b3fd8 [ 601.448316] [ 601.448316] ffff88080b0b3fd8 [ 601.448317] ffff88080b0b3fd8 [ 601.448318] ffff88080b0a8000 [ 601.448318] ffff8800369f2260 [ 601.448319] [ 601.448319] ffff8800369f2170 [ 601.448320] ffff8800369f2198 [ 601.448320] ffff8800369f2268 [ 601.448321] 0000000000000000 [ 601.448321] [ 601.448321] Call Trace: [ 601.448322] [ 601.448324] [] schedule+0x29/0x70 [ 601.448326] [ 601.448333] [] cv_wait_common+0x125/0x150 [spl] [ 601.448334] [ 601.448336] [] ? wake_up_atomic_t+0x30/0x30 [ 601.448337] [ 601.448342] [] __cv_wait+0x15/0x20 [spl] [ 601.448343] [ 601.448378] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.448379] [ 601.448411] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.448412] [ 601.448418] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.448419] [ 601.448427] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 601.448428] [ 601.448436] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 601.448437] [ 601.448445] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 601.448446] [ 601.448505] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 601.448506] [ 601.448562] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 601.448563] [ 601.448566] [] ? update_curr+0x104/0x190 [ 601.448566] [ 601.448569] [] ? account_entity_dequeue+0xae/0xd0 [ 601.448570] [ 601.448572] [] ? dequeue_entity+0x11c/0x5d0 [ 601.448574] [ 601.448615] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [ 601.448616] [ 601.448671] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.448672] [ 601.448722] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.448723] [ 601.448772] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.448773] [ 601.448775] [] ? wake_up_state+0x20/0x20 [ 601.448776] [ 601.448825] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.448826] [ 601.448828] [] ? __switch_to+0xd7/0x510 [ 601.448828] [ 601.448830] [] ? __schedule+0x371/0x8b0 [ 601.448831] [ 601.448880] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.448880] [ 601.448882] [] kthread+0xcf/0xe0 [ 601.448883] [ 601.448885] [] ? insert_kthread_work+0x40/0x40 [ 601.448885] [ 601.448887] [] ret_from_fork+0x58/0x90 [ 601.448888] [ 601.448890] [] ? insert_kthread_work+0x40/0x40 [ 601.448892] INFO: task ll_ost_io01_001:5767 blocked for more than 120 seconds. [ 601.448892] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.448893] ll_ost_io01_001 D [ 601.448894] 0000000000000000 [ 601.448895] 0 5767 2 0x00000080 [ 601.448895] ffff880809323868 [ 601.448896] 0000000000000046 [ 601.448896] ffff88080b0a8fd0 [ 601.448897] ffff880809323fd8 [ 601.448897] [ 601.448898] ffff880809323fd8 [ 601.448898] ffff880809323fd8 [ 601.448899] ffff88080b0a8fd0 [ 601.448899] ffff8800369f2260 [ 601.448899] [ 601.448900] ffff8800369f2170 [ 601.448900] ffff8800369f2198 [ 601.448901] ffff8800369f2268 [ 601.448902] 0000000000000000 [ 601.448902] [ 601.448902] Call Trace: [ 601.448903] [ 601.448905] [] schedule+0x29/0x70 [ 601.448907] [ 601.448914] [] cv_wait_common+0x125/0x150 [spl] [ 601.448914] [ 601.448916] [] ? wake_up_atomic_t+0x30/0x30 [ 601.448918] [ 601.448923] [] __cv_wait+0x15/0x20 [spl] [ 601.448924] [ 601.448958] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.448964] [ 601.448997] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.448997] [ 601.449004] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.449004] [ 601.449013] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 601.449014] [ 601.449022] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 601.449023] [ 601.449031] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 601.449032] [ 601.449091] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 601.449092] [ 601.449148] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 601.449149] [ 601.449152] [] ? update_curr+0x104/0x190 [ 601.449152] [ 601.449155] [] ? __enqueue_entity+0x78/0x80 [ 601.449155] [ 601.449158] [] ? enqueue_entity+0x26c/0xb60 [ 601.449159] [ 601.449201] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [ 601.449202] [ 601.449256] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.449257] [ 601.449308] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.449308] [ 601.449357] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.449358] [ 601.449360] [] ? default_wake_function+0x12/0x20 [ 601.449360] [ 601.449362] [] ? __wake_up_common+0x58/0x90 [ 601.449363] [ 601.449412] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.449412] [ 601.449414] [] ? __switch_to+0xd7/0x510 [ 601.449415] [ 601.449416] [] ? __schedule+0x370/0x8b0 [ 601.449417] [ 601.449465] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.449466] [ 601.449468] [] kthread+0xcf/0xe0 [ 601.449468] [ 601.449470] [] ? insert_kthread_work+0x40/0x40 [ 601.449471] [ 601.449473] [] ret_from_fork+0x58/0x90 [ 601.449473] [ 601.449475] [] ? insert_kthread_work+0x40/0x40 [ 601.449477] INFO: task ll_ost_io01_002:5768 blocked for more than 120 seconds. [ 601.449477] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.449478] ll_ost_io01_002 D [ 601.449479] 0000000000000000 [ 601.449480] 0 5768 2 0x00000080 [ 601.449480] ffff88080932b868 [ 601.449481] 0000000000000046 [ 601.449481] ffff88080b0a9fa0 [ 601.449482] ffff88080932bfd8 [ 601.449482] [ 601.449483] ffff88080932bfd8 [ 601.449483] ffff88080932bfd8 [ 601.449483] ffff88080b0a9fa0 [ 601.449484] ffff8800369f2260 [ 601.449484] [ 601.449485] ffff8800369f2170 [ 601.449485] ffff8800369f2198 [ 601.449486] ffff8800369f2268 [ 601.449486] 0000000000000000 [ 601.449487] [ 601.449487] Call Trace: [ 601.449488] [ 601.449489] [] schedule+0x29/0x70 [ 601.449492] [ 601.449498] [] cv_wait_common+0x125/0x150 [spl] [ 601.449499] [ 601.449501] [] ? wake_up_atomic_t+0x30/0x30 [ 601.449502] [ 601.449508] [] __cv_wait+0x15/0x20 [spl] [ 601.449509] [ 601.449543] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.449544] [ 601.449576] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.449577] [ 601.449583] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.449584] [ 601.449592] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 601.449593] [ 601.449601] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 601.449602] [ 601.449610] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 601.449611] [ 601.449670] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 601.449671] [ 601.449727] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 601.449727] [ 601.449730] [] ? update_curr+0x104/0x190 [ 601.449731] [ 601.449733] [] ? __enqueue_entity+0x78/0x80 [ 601.449734] [ 601.449736] [] ? enqueue_entity+0x26c/0xb60 [ 601.449738] [ 601.449779] [] ? target_send_reply_msg+0x170/0x170 [ptlrpc] [ 601.449780] [ 601.449835] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.449836] [ 601.449886] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.449887] [ 601.449936] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.449936] [ 601.449938] [] ? default_wake_function+0x12/0x20 [ 601.449939] [ 601.449941] [] ? __wake_up_common+0x58/0x90 [ 601.449942] [ 601.449990] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.450003] [ 601.450006] [] ? __switch_to+0xd7/0x510 [ 601.450006] [ 601.450008] [] ? __schedule+0x370/0x8b0 [ 601.450009] [ 601.450057] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.450058] [ 601.450060] [] kthread+0xcf/0xe0 [ 601.450061] [ 601.450062] [] ? insert_kthread_work+0x40/0x40 [ 601.450068] [ 601.450070] [] ret_from_fork+0x58/0x90 [ 601.450070] [ 601.450072] [] ? insert_kthread_work+0x40/0x40 [ 601.450075] INFO: task ll_ost_out00_00:5773 blocked for more than 120 seconds. [ 601.450075] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.450076] ll_ost_out00_00 D [ 601.450077] 0000000000000000 [ 601.450078] 0 5773 2 0x00000080 [ 601.450079] ffff88080976f9f0 [ 601.450079] 0000000000000046 [ 601.450080] ffff88080b0aeeb0 [ 601.450080] ffff88080976ffd8 [ 601.450081] [ 601.450081] ffff88080976ffd8 [ 601.450082] ffff88080976ffd8 [ 601.450082] ffff88080b0aeeb0 [ 601.450083] ffff8800369f2260 [ 601.450083] [ 601.450083] ffff8800369f2170 [ 601.450084] ffff8800369f2198 [ 601.450084] ffff8800369f2268 [ 601.450085] 0000000000000000 [ 601.450085] [ 601.450086] Call Trace: [ 601.450087] [ 601.450088] [] schedule+0x29/0x70 [ 601.450090] [ 601.450097] [] cv_wait_common+0x125/0x150 [spl] [ 601.450097] [ 601.450099] [] ? wake_up_atomic_t+0x30/0x30 [ 601.450100] [ 601.450106] [] __cv_wait+0x15/0x20 [spl] [ 601.450107] [ 601.450141] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.450142] [ 601.450174] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.450175] [ 601.450181] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.450182] [ 601.450239] [] out_tx_end+0x9d/0x5c0 [ptlrpc] [ 601.450240] [ 601.450295] [] out_handle+0x11e7/0x1920 [ptlrpc] [ 601.450295] [ 601.450298] [] ? update_curr+0x104/0x190 [ 601.450300] [ 601.450349] [] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] [ 601.450350] [ 601.450404] [] ? tgt_request_preprocess.isra.26+0x299/0x790 [ptlrpc] [ 601.450406] [ 601.450460] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.450461] [ 601.450510] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.450511] [ 601.450559] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.450560] [ 601.450562] [] ? default_wake_function+0x12/0x20 [ 601.450562] [ 601.450565] [] ? __wake_up_common+0x58/0x90 [ 601.450566] [ 601.450614] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.450615] [ 601.450617] [] ? __switch_to+0xd7/0x510 [ 601.450617] [ 601.450619] [] ? __schedule+0x370/0x8b0 [ 601.450620] [ 601.450668] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.450669] [ 601.450671] [] kthread+0xcf/0xe0 [ 601.450671] [ 601.450673] [] ? insert_kthread_work+0x40/0x40 [ 601.450674] [ 601.450676] [] ret_from_fork+0x58/0x90 [ 601.450676] [ 601.450678] [] ? insert_kthread_work+0x40/0x40 [ 601.450680] INFO: task ll_ost_out01_00:5775 blocked for more than 120 seconds. [ 601.450681] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.450681] ll_ost_out01_00 D [ 601.450682] 0000000000000000 [ 601.450683] 0 5775 2 0x00000080 [ 601.450683] ffff880805c8f9f0 [ 601.450684] 0000000000000046 [ 601.450684] ffff880805c80fd0 [ 601.450685] ffff880805c8ffd8 [ 601.450685] [ 601.450685] ffff880805c8ffd8 [ 601.450686] ffff880805c8ffd8 [ 601.450686] ffff880805c80fd0 [ 601.450687] ffff8800369f2260 [ 601.450687] [ 601.450688] ffff8800369f2170 [ 601.450688] ffff8800369f2198 [ 601.450689] ffff8800369f2268 [ 601.450689] 0000000000000000 [ 601.450690] [ 601.450690] Call Trace: [ 601.450691] [ 601.450693] [] schedule+0x29/0x70 [ 601.450695] [ 601.450701] [] cv_wait_common+0x125/0x150 [spl] [ 601.450702] [ 601.450704] [] ? wake_up_atomic_t+0x30/0x30 [ 601.450705] [ 601.450710] [] __cv_wait+0x15/0x20 [spl] [ 601.450711] [ 601.450745] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.450747] [ 601.450778] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.450779] [ 601.450786] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.450786] [ 601.450842] [] out_tx_end+0x9d/0x5c0 [ptlrpc] [ 601.450843] [ 601.450898] [] out_handle+0x11e7/0x1920 [ptlrpc] [ 601.450898] [ 601.450901] [] ? update_curr+0x104/0x190 [ 601.450903] [ 601.450951] [] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] [ 601.450956] [ 601.451011] [] ? tgt_request_preprocess.isra.26+0x299/0x790 [ptlrpc] [ 601.451012] [ 601.451066] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.451067] [ 601.451117] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.451117] [ 601.451166] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.451166] [ 601.451168] [] ? default_wake_function+0x12/0x20 [ 601.451169] [ 601.451171] [] ? __wake_up_common+0x58/0x90 [ 601.451172] [ 601.451220] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.451221] [ 601.451223] [] ? __switch_to+0xd7/0x510 [ 601.451223] [ 601.451225] [] ? __schedule+0x370/0x8b0 [ 601.451226] [ 601.451274] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.451275] [ 601.451277] [] kthread+0xcf/0xe0 [ 601.451277] [ 601.451279] [] ? insert_kthread_work+0x40/0x40 [ 601.451280] [ 601.451282] [] ret_from_fork+0x58/0x90 [ 601.451282] [ 601.451284] [] ? insert_kthread_work+0x40/0x40 [ 601.451286] INFO: task ll_ost_out01_00:5776 blocked for more than 120 seconds. [ 601.451286] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.451287] ll_ost_out01_00 D [ 601.451288] 0000000000000000 [ 601.451289] 0 5776 2 0x00000080 [ 601.451289] ffff880805c9b9f0 [ 601.451290] 0000000000000046 [ 601.451290] ffff880805c81fa0 [ 601.451291] ffff880805c9bfd8 [ 601.451291] [ 601.451291] ffff880805c9bfd8 [ 601.451292] ffff880805c9bfd8 [ 601.451293] ffff880805c81fa0 [ 601.451293] ffff8800369f2260 [ 601.451293] [ 601.451294] ffff8800369f2170 [ 601.451295] ffff8800369f2198 [ 601.451295] ffff8800369f2268 [ 601.451295] 0000000000000000 [ 601.451296] [ 601.451296] Call Trace: [ 601.451297] [ 601.451299] [] schedule+0x29/0x70 [ 601.451301] [ 601.451307] [] cv_wait_common+0x125/0x150 [spl] [ 601.451308] [ 601.451310] [] ? wake_up_atomic_t+0x30/0x30 [ 601.451311] [ 601.451317] [] __cv_wait+0x15/0x20 [spl] [ 601.451318] [ 601.451351] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.451353] [ 601.451384] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.451385] [ 601.451392] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.451392] [ 601.451448] [] out_tx_end+0x9d/0x5c0 [ptlrpc] [ 601.451449] [ 601.451503] [] out_handle+0x11e7/0x1920 [ptlrpc] [ 601.451504] [ 601.451507] [] ? update_curr+0x104/0x190 [ 601.451508] [ 601.451556] [] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] [ 601.451557] [ 601.451612] [] ? tgt_request_preprocess.isra.26+0x299/0x790 [ptlrpc] [ 601.451613] [ 601.451667] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.451668] [ 601.451717] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.451718] [ 601.451767] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.451767] [ 601.451769] [] ? default_wake_function+0x12/0x20 [ 601.451770] [ 601.451772] [] ? __wake_up_common+0x58/0x90 [ 601.451773] [ 601.451821] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.451822] [ 601.451824] [] ? __switch_to+0xd7/0x510 [ 601.451824] [ 601.451826] [] ? __schedule+0x370/0x8b0 [ 601.451827] [ 601.451875] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.451876] [ 601.451878] [] kthread+0xcf/0xe0 [ 601.451878] [ 601.451880] [] ? insert_kthread_work+0x40/0x40 [ 601.451881] [ 601.451883] [] ret_from_fork+0x58/0x90 [ 601.451883] [ 601.451885] [] ? insert_kthread_work+0x40/0x40 [ 601.451917] INFO: task txg_quiesce:7474 blocked for more than 120 seconds. [ 601.451918] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.451919] txg_quiesce D [ 601.451920] 0000000000000000 [ 601.451922] 0 7474 2 0x00000080 [ 601.451923] ffff8808203c3d38 [ 601.451924] 0000000000000046 [ 601.451924] ffff88081f3f8000 [ 601.451925] ffff8808203c3fd8 [ 601.451926] [ 601.451926] ffff8808203c3fd8 [ 601.451927] ffff8808203c3fd8 [ 601.451928] ffff88081f3f8000 [ 601.451928] ffff8803fc6016e8 [ 601.451929] [ 601.451930] ffff8803fc601668 [ 601.451930] ffff8803fc601690 [ 601.451931] ffff8803fc6016f0 [ 601.451932] 0000000000000000 [ 601.451932] [ 601.451933] Call Trace: [ 601.451934] [ 601.451937] [] schedule+0x29/0x70 [ 601.451940] [ 601.451948] [] cv_wait_common+0x125/0x150 [spl] [ 601.451986] [ 601.451988] [] ? wake_up_atomic_t+0x30/0x30 [ 601.451990] [ 601.451998] [] __cv_wait+0x15/0x20 [spl] [ 601.451999] [ 601.452058] [] txg_quiesce_thread+0x2fb/0x410 [zfs] [ 601.452060] [ 601.452114] [] ? txg_init+0x2b0/0x2b0 [zfs] [ 601.452116] [ 601.452123] [] thread_generic_wrapper+0x71/0x80 [spl] [ 601.452124] [ 601.452131] [] ? __thread_exit+0x20/0x20 [spl] [ 601.452131] [ 601.452134] [] kthread+0xcf/0xe0 [ 601.452135] [ 601.452137] [] ? insert_kthread_work+0x40/0x40 [ 601.452137] [ 601.452140] [] ret_from_fork+0x58/0x90 [ 601.452141] [ 601.452143] [] ? insert_kthread_work+0x40/0x40 [ 601.452148] INFO: task ll_ost01_005:9030 blocked for more than 120 seconds. [ 601.452149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 601.452150] ll_ost01_005 D [ 601.452150] 0000000000000000 [ 601.452152] 0 9030 2 0x00000080 [ 601.452152] ffff8803f96c39d8 [ 601.452153] 0000000000000046 [ 601.452154] ffff88040e639fa0 [ 601.452155] ffff8803f96c3fd8 [ 601.452155] [ 601.452156] ffff8803f96c3fd8 [ 601.452156] ffff8803f96c3fd8 [ 601.452157] ffff88040e639fa0 [ 601.452157] ffff8800369f2260 [ 601.452158] [ 601.452158] ffff8800369f2170 [ 601.452159] ffff8800369f2198 [ 601.452159] ffff8800369f2268 [ 601.452160] 0000000000000000 [ 601.452160] [ 601.452161] Call Trace: [ 601.452162] [ 601.452164] [] schedule+0x29/0x70 [ 601.452166] [ 601.452173] [] cv_wait_common+0x125/0x150 [spl] [ 601.452174] [ 601.452177] [] ? wake_up_atomic_t+0x30/0x30 [ 601.452178] [ 601.452185] [] __cv_wait+0x15/0x20 [spl] [ 601.452186] [ 601.452227] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 601.452228] [ 601.452268] [] dmu_tx_assign+0x91/0x490 [zfs] [ 601.452269] [ 601.452277] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 601.452278] [ 601.452289] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 601.452290] [ 601.452301] [] ofd_destroy+0x5b0/0x940 [ofd] [ 601.452301] [ 601.452311] [] ofd_destroy_by_fid+0x1e3/0x4a0 [ofd] [ 601.452312] [ 601.452367] [] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [ 601.452368] [ 601.452421] [] ? ldlm_expired_completion_wait+0x240/0x240 [ptlrpc] [ 601.452422] [ 601.452431] [] ofd_destroy_hdl+0x267/0x970 [ofd] [ 601.452433] [ 601.452502] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 601.452503] [ 601.452565] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 601.452566] [ 601.452628] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 601.452628] [ 601.452631] [] ? default_wake_function+0x12/0x20 [ 601.452631] [ 601.452634] [] ? __wake_up_common+0x58/0x90 [ 601.452636] [ 601.452696] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 601.452697] [ 601.452699] [] ? __switch_to+0xd7/0x510 [ 601.452700] [ 601.452702] [] ? __schedule+0x370/0x8b0 [ 601.452704] [ 601.452764] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [ 601.452764] [ 601.452767] [] kthread+0xcf/0xe0 [ 601.452768] [ 601.452770] [] ? insert_kthread_work+0x40/0x40 [ 601.452771] [ 601.452773] [] ret_from_fork+0x58/0x90 [ 601.452774] [ 601.452776] [] ? insert_kthread_work+0x40/0x40 [ 625.805846] sched: RT throttling activated [ 662.884028] LNet: Service thread pid 75988 was inactive for 200.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 663.090474] Pid: 75988, comm: ll_ost_io01_033 [ 663.144281] Call Trace: [ 663.194479] [] schedule+0x29/0x70 [ 663.255674] [] cv_wait_common+0x125/0x150 [spl] [ 663.331307] [] ? autoremove_wake_function+0x0/0x40 [ 663.410082] [] __cv_wait+0x15/0x20 [spl] [ 663.478518] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 663.551100] [] dmu_tx_assign+0x91/0x490 [zfs] [ 663.624664] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 663.704490] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 663.779093] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 663.859958] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 663.933602] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 664.010313] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 664.089017] [] ? update_curr+0x104/0x190 [ 664.157323] [] ? __enqueue_entity+0x78/0x80 [ 664.228765] [] ? enqueue_entity+0x26c/0xb60 [ 664.300206] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 664.383135] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 664.467094] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 664.560435] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 664.643311] [] ? default_wake_function+0x12/0x20 [ 664.719959] [] ? __wake_up_common+0x58/0x90 [ 664.791476] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 664.868138] [] ? __switch_to+0xd7/0x510 [ 664.935454] [] ? __schedule+0x370/0x8b0 [ 665.002811] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 665.079490] [] kthread+0xcf/0xe0 [ 665.139501] [] ? kthread+0x0/0xe0 [ 665.200536] [] ret_from_fork+0x58/0x90 [ 665.266767] [] ? kthread+0x0/0xe0 [ 665.327768] [ 665.347136] LustreError: dumping log to /tmp/lustre-log.1513210785.75988 [ 665.961980] LNet: Service thread pid 22795 was inactive for 200.25s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 666.168441] Pid: 22795, comm: ll_ost_out00_01 [ 666.222290] Call Trace: [ 666.272541] [] schedule+0x29/0x70 [ 666.333674] [] cv_wait_common+0x125/0x150 [spl] [ 666.409378] [] ? autoremove_wake_function+0x0/0x40 [ 666.488219] [] __cv_wait+0x15/0x20 [spl] [ 666.556684] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 666.629298] [] dmu_tx_assign+0x91/0x490 [zfs] [ 666.702940] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 666.782884] [] out_tx_end+0x9d/0x5c0 [ptlrpc] [ 666.856518] [] out_handle+0x11e7/0x1920 [ptlrpc] [ 666.933168] [] ? update_curr+0x104/0x190 [ 667.001526] [] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] [ 667.084525] [] ? tgt_request_preprocess.isra.26+0x299/0x790 [ptlrpc] [ 667.182086] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 667.266141] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 667.359550] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 667.442483] [] ? default_wake_function+0x12/0x20 [ 667.519202] [] ? __wake_up_common+0x58/0x90 [ 667.590746] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 667.667484] [] ? __switch_to+0xd7/0x510 [ 667.734807] [] ? __schedule+0x370/0x8b0 [ 667.802222] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 667.878919] [] kthread+0xcf/0xe0 [ 667.938973] [] ? kthread+0x0/0xe0 [ 668.000095] [] ret_from_fork+0x58/0x90 [ 668.066387] [] ? kthread+0x0/0xe0 [ 668.127495] [ 668.146998] LNet: Service thread pid 42781 was inactive for 201.80s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 668.353214] Pid: 42781, comm: ll_ost01_035 [ 668.404019] Call Trace: [ 668.454405] [] schedule+0x29/0x70 [ 668.515586] [] schedule_timeout+0x174/0x2c0 [ 668.587152] [] ? process_timeout+0x0/0x10 [ 668.656674] [] ? ldlm_expired_completion_wait+0x0/0x240 [ptlrpc] [ 668.750131] [] ldlm_completion_ast+0x5b1/0x920 [ptlrpc] [ 668.834205] [] ? default_wake_function+0x0/0x20 [ 668.909959] [] ldlm_cli_enqueue_local+0x230/0x860 [ptlrpc] [ 668.997229] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [ 669.081413] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [ 669.163467] [] ofd_destroy_by_fid+0x1cc/0x4a0 [ofd] [ 669.243473] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [ 669.325611] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [ 669.409676] [] ofd_destroy_hdl+0x267/0x970 [ofd] [ 669.486512] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 669.570585] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 669.664025] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 669.747000] [] ? default_wake_function+0x0/0x20 [ 669.822765] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 669.899551] [] ? __switch_to+0xd7/0x510 [ 669.966940] [] ? __schedule+0x370/0x8b0 [ 670.034395] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 670.111161] [] kthread+0xcf/0xe0 [ 670.171279] [] ? kthread+0x0/0xe0 [ 670.232447] [] ret_from_fork+0x58/0x90 [ 670.298739] [] ? kthread+0x0/0xe0 [ 670.359868] [ 670.379370] LNet: Service thread pid 5775 was inactive for 200.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 670.584571] Pid: 5775, comm: ll_ost_out01_00 [ 670.637474] Call Trace: [ 670.687855] [] schedule+0x29/0x70 [ 670.749023] [] cv_wait_common+0x125/0x150 [spl] [ 670.824712] [] ? autoremove_wake_function+0x0/0x40 [ 670.903555] [] __cv_wait+0x15/0x20 [spl] [ 670.972043] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 671.044687] [] dmu_tx_assign+0x91/0x490 [zfs] [ 671.118358] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 671.198303] [] out_tx_end+0x9d/0x5c0 [ptlrpc] [ 671.272046] [] out_handle+0x11e7/0x1920 [ptlrpc] [ 671.348813] [] ? update_curr+0x104/0x190 [ 671.417334] [] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc] [ 671.500455] [] ? tgt_request_preprocess.isra.26+0x299/0x790 [ptlrpc] [ 671.598112] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 671.682197] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 671.775615] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 671.858609] [] ? default_wake_function+0x12/0x20 [ 671.935399] [] ? __wake_up_common+0x58/0x90 [ 672.007034] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 672.083830] [] ? __switch_to+0xd7/0x510 [ 672.151267] [] ? __schedule+0x370/0x8b0 [ 672.218723] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 672.295468] [] kthread+0xcf/0xe0 [ 672.355568] [] ? kthread+0x0/0xe0 [ 672.416691] [] ret_from_fork+0x58/0x90 [ 672.483003] [] ? kthread+0x0/0xe0 [ 672.544110] [ 672.563570] Pid: 179417, comm: ll_ost_io00_054 [ 672.618296] Call Trace: [ 672.668506] [] schedule+0x29/0x70 [ 672.729526] [] cv_wait_common+0x125/0x150 [spl] [ 672.805151] [] ? autoremove_wake_function+0x0/0x40 [ 672.883897] [] __cv_wait+0x15/0x20 [spl] [ 672.952274] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 673.024796] [] dmu_tx_assign+0x91/0x490 [zfs] [ 673.098339] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 673.178115] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 673.252645] [] ofd_object_punch+0x70a/0xd20 [ofd] [ 673.330313] [] ofd_punch_hdl+0x471/0xaa0 [ofd] [ 673.404947] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 673.488962] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 673.582363] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 673.665309] [] ? default_wake_function+0x12/0x20 [ 673.741967] [] ? __wake_up_common+0x58/0x90 [ 673.813398] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 673.889959] [] ? __switch_to+0xd7/0x510 [ 673.957190] [] ? __schedule+0x371/0x8b0 [ 674.024444] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 674.101058] [] kthread+0xcf/0xe0 [ 674.161016] [] ? kthread+0x0/0xe0 [ 674.222005] [] ret_from_fork+0x58/0x90 [ 674.288191] [] ? kthread+0x0/0xe0 [ 674.349118] [ 674.368425] LNet: Service thread pid 23227 was inactive for 204.31s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 702.819273] LNet: Service thread pid 81590 was inactive for 200.73s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 702.977030] LNet: Skipped 109 previous similar messages [ 703.041376] LustreError: dumping log to /tmp/lustre-log.1513210823.81590 [ 725.858880] LNet: Service thread pid 27489 was inactive for 200.27s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 726.016567] LustreError: dumping log to /tmp/lustre-log.1513210845.27489 [ 726.370888] LustreError: dumping log to /tmp/lustre-log.1513210846.8479 [ 764.390228] LustreError: 27491:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1513210584, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-soaked-OST0005_UUID lock: ffff8807d487a800/0xd3772cc93cf96406 lrc: 3/0,1 mode: --/PW res: [0x1779539:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 27491 timeout: 0 lvb_type: 0 [ 764.390464] LustreError: dumping log to /tmp/lustre-log.1513210884.5757 [ 764.997303] LustreError: 27491:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 4 previous similar messages [ 766.341192] LustreError: 42781:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1513210586, 300s ago); not entering recovery in server code, just going back to sleep ns: filter-soaked-OST0005_UUID lock: ffff8807d845e200/0xd3772cc93cf969d2 lrc: 3/0,1 mode: --/PW res: [0x440000400:0x8a1f07:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x40010080000000 nid: local remote: 0x0 expref: -99 pid: 42781 timeout: 0 lvb_type: 0 [ 766.875415] LustreError: 42781:0:(ldlm_request.c:130:ldlm_expired_completion_wait()) Skipped 1 previous similar message [ 923.999500] LNet: Service thread pid 26752 was inactive for 312.93s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 924.046768] LNet: Skipped 14 previous similar messages [ 924.066097] LustreError: dumping log to /tmp/lustre-log.1513211044.26752 [ 1057.257248] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807edd62850 x1586709579097600/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:167/0 lens 520/4320 e 21 to 0 dl 1513211182 ref 2 fl Interpret:/0/0 rc 0/0 [ 1057.298459] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages [ 1057.821214] Lustre: 179234:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8800a0a9e850 x1586704107406384/t0(0) o4->6943d9de-cea7-4a53-1661-77d616b076c6@192.168.1.136@o2ib:167/0 lens 608/448 e 23 to 0 dl 1513211182 ref 2 fl Interpret:/0/0 rc 0/0 [ 1057.865142] Lustre: 179234:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 6 previous similar messages [ 1058.823183] Lustre: 179234:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803f7367450 x1586704140058704/t0(0) o4->13c3c95f-3b0c-4a15-dbb3-8d98d51a2eaf@192.168.1.139@o2ib:168/0 lens 608/448 e 23 to 0 dl 1513211183 ref 2 fl Interpret:/0/0 rc 0/0 [ 1058.867464] Lustre: 179234:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 71 previous similar messages [ 1061.287152] Lustre: 5774:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803c5867050 x1586709551824368/t0(0) o1000->soaked-MDT0002-mdtlov_UUID@192.168.1.110@o2ib:171/0 lens 368/0 e 21 to 0 dl 1513211186 ref 2 fl New:/0/ffffffff rc 0/-1 [ 1061.331571] Lustre: 5774:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 24 previous similar messages [ 1063.215300] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 1063.230619] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 1063.246162] Lustre: Skipped 8 previous similar messages [ 1063.809089] Lustre: soaked-OST0005: Client ef49c3a2-4ffa-13ce-cda7-f57f5ed67b0d (at 192.168.1.132@o2ib) reconnecting [ 1063.823924] Lustre: Skipped 3 previous similar messages [ 1063.904271] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334594 [ 1064.693968] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629665 [ 1065.295084] Lustre: 5774:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803e07c0850 x1586709552004976/t0(0) o1000->soaked-MDT0002-mdtlov_UUID@192.168.1.110@o2ib:175/0 lens 368/0 e 21 to 0 dl 1513211190 ref 2 fl New:/0/ffffffff rc 0/-1 [ 1065.340251] Lustre: 5774:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 72 previous similar messages [ 1065.595807] Lustre: soaked-OST0005: Client 9d9e7df2-293b-033f-3d78-e6db9c11dd73 (at 192.168.1.117@o2ib) reconnecting [ 1065.611198] Lustre: Skipped 7 previous similar messages [ 1065.620584] Lustre: soaked-OST0005: Connection restored to 9d9e7df2-293b-033f-3d78-e6db9c11dd73 (at 192.168.1.117@o2ib) [ 1065.636188] Lustre: Skipped 11 previous similar messages [ 1065.695493] Lustre: soaked-OST0005: deleting orphan objects from 0x0:24614878 to 0x0:24626116 [ 1068.555367] Lustre: soaked-OST000b: Client soaked-MDT0002-mdtlov_UUID (at 192.168.1.110@o2ib) reconnecting [ 1068.569644] Lustre: Skipped 3 previous similar messages [ 1068.579898] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10621857 [ 1068.821475] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052114 [ 1069.542521] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610081 [ 1069.542863] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608161 [ 1072.079190] Lustre: soaked-OST0011: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [ 1072.080904] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036209 [ 1072.106708] Lustre: Skipped 8 previous similar messages [ 1072.115194] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273537 [ 1073.261098] Lustre: soaked-OST0005: Client a1998523-d187-abef-1227-02d3e3ca69ad (at 192.168.1.121@o2ib) reconnecting [ 1073.276383] Lustre: Skipped 5 previous similar messages [ 1074.292925] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88080edb5850 x1586709314039488/t0(0) o1000->soaked-MDT0001-mdtlov_UUID@192.168.1.109@o2ib:184/0 lens 368/0 e 21 to 0 dl 1513211199 ref 2 fl New:/0/ffffffff rc 0/-1 [ 1074.336514] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 174 previous similar messages [ 1074.380802] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028097 [ 1074.381162] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032306 [ 1090.330640] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88082c69d050 x1586709314184432/t0(0) o1000->soaked-MDT0001-mdtlov_UUID@192.168.1.109@o2ib:200/0 lens 368/0 e 11 to 0 dl 1513211215 ref 2 fl New:/0/ffffffff rc 0/-1 [ 1090.375088] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages [ 1127.012629] Lustre: soaked-OST0005: Client 256168e3-cb16-01bb-1477-b14e2e4ce2a5 (at 192.168.1.126@o2ib) reconnecting [ 1127.027710] Lustre: Skipped 2 previous similar messages [ 1127.036820] Lustre: soaked-OST0005: Connection restored to 256168e3-cb16-01bb-1477-b14e2e4ce2a5 (at 192.168.1.126@o2ib) [ 1127.052104] Lustre: Skipped 3 previous similar messages [ 1206.106676] Lustre: 136033:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807c6eb7c50 x1586704303318400/t0(0) o4->9d9e7df2-293b-033f-3d78-e6db9c11dd73@192.168.1.117@o2ib:316/0 lens 608/448 e 4 to 0 dl 1513211331 ref 2 fl Interpret:H/0/0 rc 0/0 [ 1206.214787] Lustre: 136033:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 37 previous similar messages [ 1305.388932] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [ 1658.194889] Lustre: 136033:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807f3415050 x1586704164660976/t0(0) o4->627621b7-cb99-7763-2256-7aec3c9afc32@192.168.1.138@o2ib:13/0 lens 608/448 e 1 to 0 dl 1513211783 ref 2 fl Interpret:/0/0 rc 0/0 [ 1658.302892] Lustre: 136033:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 21 previous similar messages [ 1664.204559] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 1664.242800] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 1664.281673] Lustre: Skipped 1 previous similar message [ 1664.931872] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334626 [ 1665.476464] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629697 [ 1666.635969] Lustre: soaked-OST0005: deleting orphan objects from 0x0:24614878 to 0x0:24626148 [ 1669.370583] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052146 [ 1669.476923] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10621889 [ 1670.478035] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610113 [ 1670.478314] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608193 [ 1672.371048] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036241 [ 1673.129948] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273569 [ 1675.372134] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028129 [ 1675.372691] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032338 [ 1728.555222] Lustre: soaked-OST0005: Client 256168e3-cb16-01bb-1477-b14e2e4ce2a5 (at 192.168.1.126@o2ib) reconnecting [ 1728.592966] Lustre: Skipped 24 previous similar messages [ 1728.613524] Lustre: soaked-OST0005: Connection restored to 256168e3-cb16-01bb-1477-b14e2e4ce2a5 (at 192.168.1.126@o2ib) [ 1728.651887] Lustre: Skipped 22 previous similar messages [ 2260.296553] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8807eba76450 x1586709579120832/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:614/0 lens 520/0 e 0 to 0 dl 1513212384 ref 2 fl New:/2/ffffffff rc 0/-1 [ 2260.340342] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 438 previous similar messages [ 2265.194133] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 2265.209162] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 2265.903959] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334658 [ 2266.440532] LNet: Service thread pid 131596 was inactive for 1201.18s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 2266.466016] LNet: Skipped 1 previous similar message [ 2266.473975] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629729 [ 2266.488314] Pid: 131596, comm: ll_ost_io00_093 [ 2266.495828] Call Trace: [ 2266.505182] [] schedule+0x29/0x70 [ 2266.513035] [] cv_wait_common+0x125/0x150 [spl] [ 2266.522231] [] ? autoremove_wake_function+0x0/0x40 [ 2266.531538] [] __cv_wait+0x15/0x20 [spl] [ 2266.539754] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2266.548317] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2266.556806] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2266.565876] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2266.574307] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2266.583287] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2266.591535] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2266.600044] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2266.608563] [] ? task_rq_unlock+0x1a/0x20 [ 2266.616241] [] ? __sched_setscheduler+0x882/0x8c0 [ 2266.624539] [] ? __enqueue_entity+0x78/0x80 [ 2266.632251] [] ? enqueue_entity+0x26c/0xb60 [ 2266.639931] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2266.648663] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2266.657416] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2266.667059] [] ? ptlrpc_start_thread+0x26d/0xbd0 [ptlrpc] [ 2266.675967] [] ? __wake_up_common+0x58/0x90 [ 2266.683567] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2266.691534] [] ? __switch_to+0xd7/0x510 [ 2266.698631] [] ? __schedule+0x370/0x8b0 [ 2266.705723] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2266.713660] [] kthread+0xcf/0xe0 [ 2266.720043] [] ? kthread+0x0/0xe0 [ 2266.726535] [] ret_from_fork+0x58/0x90 [ 2266.733508] [] ? kthread+0x0/0xe0 [ 2266.739978] [ 2266.742541] LustreError: dumping log to /tmp/lustre-log.1513212386.131596 [ 2266.817006] Pid: 131595, comm: ll_ost_io00_092 [ 2266.823659] Call Trace: [ 2266.830901] [] schedule+0x29/0x70 [ 2266.837827] [] cv_wait_common+0x125/0x150 [spl] [ 2266.845867] [] ? autoremove_wake_function+0x0/0x40 [ 2266.854114] [] __cv_wait+0x15/0x20 [spl] [ 2266.861354] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2266.868909] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2266.876521] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2266.884767] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2266.892420] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2266.900664] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2266.908208] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2266.916017] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2266.923931] [] ? task_rq_unlock+0x1a/0x20 [ 2266.931008] [] ? __sched_setscheduler+0x882/0x8c0 [ 2266.938810] [] ? sched_set_stop_task+0x80/0xb0 [ 2266.946353] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2266.954656] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2266.963091] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2266.972324] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 2266.980852] [] ? kmalloc_order_trace+0x2e/0xa0 [ 2266.988346] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2266.996010] [] ? __switch_to+0xd7/0x510 [ 2267.002799] [] ? __schedule+0x39d/0x8b0 [ 2267.009620] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2267.017287] [] kthread+0xcf/0xe0 [ 2267.023398] [] ? kthread+0x0/0xe0 [ 2267.029619] [] ret_from_fork+0x58/0x90 [ 2267.036306] [] ? kthread+0x0/0xe0 [ 2267.042514] [ 2267.044841] LNet: Service thread pid 131594 was inactive for 1202.68s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 2267.065366] LNet: Skipped 1 previous similar message [ 2267.071614] Pid: 131594, comm: ll_ost_io00_091 [ 2267.077273] Call Trace: [ 2267.083004] [] schedule+0x29/0x70 [ 2267.089239] [] cv_wait_common+0x125/0x150 [spl] [ 2267.096841] [] ? autoremove_wake_function+0x0/0x40 [ 2267.104742] [] __cv_wait+0x15/0x20 [spl] [ 2267.111686] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2267.119026] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2267.126435] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2267.134444] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2267.141967] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2267.150059] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2267.157484] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2267.165201] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2267.173077] [] ? task_rq_unlock+0x1a/0x20 [ 2267.180089] [] ? __sched_setscheduler+0x882/0x8c0 [ 2267.187884] [] ? sched_set_stop_task+0x80/0xb0 [ 2267.195397] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2267.203711] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2267.212131] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2267.221401] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 2267.229992] [] ? kmalloc_order_trace+0x2e/0xa0 [ 2267.237549] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2267.245268] [] ? __switch_to+0xd7/0x510 [ 2267.252105] [] ? __schedule+0x39d/0x8b0 [ 2267.258993] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2267.266728] [] kthread+0xcf/0xe0 [ 2267.272905] [] ? kthread+0x0/0xe0 [ 2267.279165] [] ret_from_fork+0x58/0x90 [ 2267.285912] [] ? kthread+0x0/0xe0 [ 2267.292164] [ 2267.294519] Pid: 131593, comm: ll_ost_io00_090 [ 2267.300181] Call Trace: [ 2267.305909] [] schedule+0x29/0x70 [ 2267.312141] [] cv_wait_common+0x125/0x150 [spl] [ 2267.319719] [] ? autoremove_wake_function+0x0/0x40 [ 2267.327614] [] __cv_wait+0x15/0x20 [spl] [ 2267.334560] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2267.341883] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2267.349292] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2267.357276] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2267.364806] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2267.372896] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2267.380316] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2267.387998] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2267.395827] [] ? task_rq_unlock+0x1a/0x20 [ 2267.402789] [] ? __sched_setscheduler+0x882/0x8c0 [ 2267.410531] [] ? sched_set_stop_task+0x80/0xb0 [ 2267.418004] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2267.426266] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2267.434629] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2267.443871] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 2267.452386] [] ? kmalloc_order_trace+0x2e/0xa0 [ 2267.459887] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2267.467555] [] ? __switch_to+0xd7/0x510 [ 2267.474339] [] ? __schedule+0x39d/0x8b0 [ 2267.481157] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2267.488818] [] kthread+0xcf/0xe0 [ 2267.494939] [] ? finish_task_switch+0x54/0x160 [ 2267.502426] [] ? kthread+0x0/0xe0 [ 2267.508610] [] ret_from_fork+0x58/0x90 [ 2267.515260] [] ? kthread+0x0/0xe0 [ 2267.521472] [ 2267.523756] Pid: 131592, comm: ll_ost_io00_089 [ 2267.529341] Call Trace: [ 2267.534890] [] schedule+0x29/0x70 [ 2267.541022] [] cv_wait_common+0x125/0x150 [spl] [ 2267.548492] [] ? autoremove_wake_function+0x0/0x40 [ 2267.556255] [] __cv_wait+0x15/0x20 [spl] [ 2267.563034] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2267.570212] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2267.576495] Lustre: soaked-OST0005: deleting orphan objects from 0x0:24614878 to 0x0:24626180 [ 2267.587551] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2267.595419] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2267.602776] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2267.610750] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2267.618040] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2267.625618] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2267.633380] [] ? task_rq_unlock+0x1a/0x20 [ 2267.640259] [] ? __sched_setscheduler+0x882/0x8c0 [ 2267.647911] [] ? sched_set_stop_task+0x80/0xb0 [ 2267.655289] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2267.663458] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2267.671723] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2267.680878] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 2267.689324] [] ? kmalloc_order_trace+0x2e/0xa0 [ 2267.696727] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2267.704300] [] ? __switch_to+0xd7/0x510 [ 2267.710996] [] ? __schedule+0x39d/0x8b0 [ 2267.717717] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2267.725293] [] kthread+0xcf/0xe0 [ 2267.731311] [] ? kthread+0x0/0xe0 [ 2267.737434] [] ret_from_fork+0x58/0x90 [ 2267.744036] [] ? kthread+0x0/0xe0 [ 2267.750152] [ 2267.752377] LNet: Service thread pid 131591 was inactive for 1203.38s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 2267.768122] LNet: Skipped 1 previous similar message [ 2270.312010] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052178 [ 2270.441736] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10621921 [ 2270.536366] LNet: Service thread pid 132149 was inactive for 1203.39s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 2270.553642] LNet: Skipped 78 previous similar messages [ 2270.560679] LustreError: dumping log to /tmp/lustre-log.1513212390.132149 [ 2271.442508] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610145 [ 2271.442947] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608225 [ 2273.312510] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036273 [ 2274.119556] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273601 [ 2276.312604] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028161 [ 2276.313109] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032370 [ 2327.879425] LNet: Service thread pid 134361 was inactive for 1200.81s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 2327.897787] LustreError: dumping log to /tmp/lustre-log.1513212447.134361 [ 2832.362703] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [ 2860.350248] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880806d41450 x1586704091559456/t0(0) o4->8a80e4dc-a534-30af-d839-075339342992@192.168.1.130@o2ib:460/0 lens 608/448 e 0 to 0 dl 1513212985 ref 2 fl Interpret:/2/0 rc 0/0 [ 2860.350252] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807ff595450 x1586709579120832/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:460/0 lens 520/0 e 0 to 0 dl 1513212985 ref 2 fl New:/2/ffffffff rc 0/-1 [ 2860.350259] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 308 previous similar messages [ 2861.173286] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages [ 2864.446176] LNet: Service thread pid 136821 was inactive for 1200.13s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 2864.656201] LNet: Skipped 2 previous similar messages [ 2864.719152] Pid: 136821, comm: ll_ost_io00_100 [ 2864.774893] Call Trace: [ 2864.826881] [] schedule+0x29/0x70 [ 2864.888783] [] rwsem_down_read_failed+0x10d/0x1a0 [ 2864.967409] [] call_rwsem_down_read_failed+0x18/0x30 [ 2865.049087] [] down_read+0x20/0x40 [ 2865.112048] [] osd_read_lock+0x39/0x40 [osd_zfs] [ 2865.189631] [] ofd_preprw_write.isra.30+0xc6/0xd80 [ofd] [ 2865.275554] [] ofd_preprw+0x422/0x1150 [ofd] [ 2865.349007] [] tgt_brw_write+0xc34/0x17c0 [ptlrpc] [ 2865.428686] [] ? task_rq_unlock+0x1a/0x20 [ 2865.498965] [] ? __enqueue_entity+0x78/0x80 [ 2865.571368] [] ? enqueue_entity+0x26c/0xb60 [ 2865.643755] [] ? enqueue_task_fair+0x208/0x6c0 [ 2865.719247] [] ? check_preempt_curr+0x78/0xa0 [ 2865.793649] [] ? mutex_lock+0x12/0x2f [ 2865.859763] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2865.944633] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2866.038789] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 2866.125614] [] ? kmalloc_order_trace+0x2e/0xa0 [ 2866.183574] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 2866.183577] Lustre: Skipped 25 previous similar messages [ 2866.183624] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 2866.183626] Lustre: Skipped 25 previous similar messages [ 2866.593030] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2866.670551] [] ? __switch_to+0xd7/0x510 [ 2866.738680] [] ? __schedule+0x39d/0x8b0 [ 2866.806860] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2866.845612] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334690 [ 2867.003609] [] kthread+0xcf/0xe0 [ 2867.064527] [] ? kthread+0x0/0xe0 [ 2867.126437] [] ret_from_fork+0x58/0x90 [ 2867.193519] [] ? kthread+0x0/0xe0 [ 2867.255350] [ 2867.275500] LustreError: dumping log to /tmp/lustre-log.1513212987.136821 [ 2867.414185] LNet: Service thread pid 136812 was inactive for 1203.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 2867.536213] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629761 [ 2867.744645] Pid: 136812, comm: ll_ost_io01_076 [ 2867.800115] Call Trace: [ 2867.851405] [] schedule+0x29/0x70 [ 2867.912960] [] cv_wait_common+0x125/0x150 [spl] [ 2867.989023] [] ? autoremove_wake_function+0x0/0x40 [ 2868.068179] [] __cv_wait+0x15/0x20 [spl] [ 2868.136951] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2868.209758] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2868.283562] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2868.363677] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2868.438616] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2868.519779] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2868.557593] Lustre: soaked-OST0005: deleting orphan objects from 0x0:24614878 to 0x0:24626212 [ 2868.697801] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2868.774858] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2868.853902] [] ? task_rq_unlock+0x1a/0x20 [ 2868.923587] [] ? __sched_setscheduler+0x882/0x8c0 [ 2869.001678] [] ? sched_set_stop_task+0x80/0xb0 [ 2869.076690] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2869.160057] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2869.244457] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2869.338277] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 2869.424706] [] ? kmalloc_order_trace+0x2e/0xa0 [ 2869.499734] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2869.576831] [] ? __switch_to+0xd7/0x510 [ 2869.644537] [] ? __schedule+0x39d/0x8b0 [ 2869.712272] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2869.789303] [] kthread+0xcf/0xe0 [ 2869.849692] [] ? kthread+0x0/0xe0 [ 2869.911066] [] ret_from_fork+0x58/0x90 [ 2869.977622] [] ? kthread+0x0/0xe0 [ 2870.038903] [ 2870.058546] Pid: 136834, comm: ll_ost_io00_113 [ 2870.113370] Call Trace: [ 2870.163861] [] schedule+0x29/0x70 [ 2870.225086] [] cv_wait_common+0x125/0x150 [spl] [ 2870.300830] [] ? autoremove_wake_function+0x0/0x40 [ 2870.379702] [] __cv_wait+0x15/0x20 [spl] [ 2870.448209] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2870.520861] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2870.594536] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2870.674462] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2870.749162] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2870.830037] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2870.903660] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2870.980344] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2871.058982] [] ? task_rq_unlock+0x1a/0x20 [ 2871.128280] [] ? __enqueue_entity+0x78/0x80 [ 2871.199658] [] ? enqueue_entity+0x26c/0xb60 [ 2871.270197] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052210 [ 2871.389364] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2871.472303] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2871.536304] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10621953 [ 2871.676735] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2871.770240] [] ? ptlrpc_start_thread+0x26d/0xbd0 [ptlrpc] [ 2871.856368] [] ? __wake_up_common+0x58/0x90 [ 2871.928015] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2872.004752] [] ? __switch_to+0xd7/0x510 [ 2872.072163] [] ? __schedule+0x370/0x8b0 [ 2872.139619] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2872.216338] [] kthread+0xcf/0xe0 [ 2872.276362] [] ? kthread+0x0/0xe0 [ 2872.337457] [] ret_from_fork+0x58/0x90 [ 2872.403744] [] ? kthread+0x0/0xe0 [ 2872.464769] [ 2872.484237] LNet: Service thread pid 136850 was inactive for 1207.08s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 2872.537294] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610177 [ 2872.537686] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608257 [ 2872.933236] LNet: Skipped 1 previous similar message [ 2872.994434] Pid: 136850, comm: ll_ost_io01_092 [ 2873.049413] Call Trace: [ 2873.099927] [] schedule+0x29/0x70 [ 2873.161191] [] cv_wait_common+0x125/0x150 [spl] [ 2873.237049] [] ? autoremove_wake_function+0x0/0x40 [ 2873.316043] [] __cv_wait+0x15/0x20 [spl] [ 2873.384781] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2873.457626] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2873.531523] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2873.611530] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2873.686261] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2873.767235] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2873.841040] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2873.917889] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2873.996808] [] ? ___slab_alloc+0x209/0x4f0 [ 2874.067439] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2874.150712] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2874.235017] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2874.235100] [] ? ptlrpc_start_thread+0x26d/0xbd0 [ptlrpc] [ 2874.235105] [] ? kmalloc_order_trace+0x2e/0xa0 [ 2874.235169] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2874.235173] [] ? __switch_to+0xd7/0x510 [ 2874.235177] [] ? __schedule+0x39d/0x8b0 [ 2874.235246] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2874.235250] [] kthread+0xcf/0xe0 [ 2874.235253] [] ? kthread+0x0/0xe0 [ 2874.235257] [] ret_from_fork+0x58/0x90 [ 2874.235261] [] ? kthread+0x0/0xe0 [ 2874.235262] [ 2874.235271] Pid: 136863, comm: ll_ost_io01_105 [ 2874.235272] Call Trace: [ 2874.235275] [] schedule+0x29/0x70 [ 2874.235294] [] cv_wait_common+0x125/0x150 [spl] [ 2874.235298] [] ? autoremove_wake_function+0x0/0x40 [ 2874.235308] [] __cv_wait+0x15/0x20 [spl] [ 2874.235366] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 2874.235416] [] dmu_tx_assign+0x91/0x490 [zfs] [ 2874.235427] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 2874.235439] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 2874.235450] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 2874.235461] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 2874.235538] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 2874.235613] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 2874.235618] [] ? task_rq_unlock+0x1a/0x20 [ 2874.235623] [] ? __sched_setscheduler+0x882/0x8c0 [ 2874.235627] [] ? __enqueue_entity+0x78/0x80 [ 2874.235631] [] ? enqueue_entity+0x26c/0xb60 [ 2874.235691] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 2874.235763] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 2874.235834] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 2874.235904] [] ? ptlrpc_start_thread+0x26d/0xbd0 [ptlrpc] [ 2874.235908] [] ? __wake_up_common+0x58/0x90 [ 2874.236002] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 2874.236008] [] ? __switch_to+0xd7/0x510 [ 2874.236012] [] ? __schedule+0x371/0x8b0 [ 2874.236102] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 2874.236106] [] kthread+0xcf/0xe0 [ 2874.236109] [] ? kthread+0x0/0xe0 [ 2874.236112] [] ret_from_fork+0x58/0x90 [ 2874.236116] [] ? kthread+0x0/0xe0 [ 2874.236118] [ 2874.236136] LNet: Service thread pid 136862 was inactive for 1206.85s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 2874.269837] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036305 [ 2875.108434] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273633 [ 2877.271003] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028193 [ 2877.271601] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032402 [ 3198.356423] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [ 3462.451911] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88026602e050 x1586704102710928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:307/0 lens 608/0 e 0 to 0 dl 1513213587 ref 2 fl New:/2/ffffffff rc 0/-1 [ 3462.553572] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 453 previous similar messages [ 3466.547824] LNet: Service thread pid 139711 was inactive for 1201.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 3466.608146] LNet: Skipped 1 previous similar message [ 3466.626379] Pid: 139711, comm: ll_ost_io01_116 [ 3466.642759] Call Trace: [ 3466.658862] [] schedule+0x29/0x70 [ 3466.676806] [] cv_wait_common+0x125/0x150 [spl] [ 3466.698602] [] ? autoremove_wake_function+0x0/0x40 [ 3466.721184] [] __cv_wait+0x15/0x20 [spl] [ 3466.740868] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 3466.761601] [] dmu_tx_assign+0x91/0x490 [zfs] [ 3466.782530] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 3466.805119] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 3466.826221] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 3466.848998] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 3466.869799] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 3466.891428] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 3466.913516] [] ? task_rq_unlock+0x1a/0x20 [ 3466.933001] [] ? __sched_setscheduler+0x882/0x8c0 [ 3466.954767] [] ? __enqueue_entity+0x78/0x80 [ 3466.974821] [] ? enqueue_entity+0x26c/0xb60 [ 3466.994886] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 3467.018104] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 3467.041628] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 3467.067733] [] ? ptlrpc_start_thread+0x26d/0xbd0 [ptlrpc] [ 3467.091770] [] ? __wake_up_common+0x58/0x90 [ 3467.111840] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 3467.133346] [] ? __switch_to+0xd7/0x510 [ 3467.152276] [] ? __schedule+0x370/0x8b0 [ 3467.156623] Lustre: soaked-OST0005: Client 8a80e4dc-a534-30af-d839-075339342992 (at 192.168.1.130@o2ib) reconnecting [ 3467.156625] Lustre: Skipped 25 previous similar messages [ 3467.156659] Lustre: soaked-OST0005: Connection restored to 8a80e4dc-a534-30af-d839-075339342992 (at 192.168.1.130@o2ib) [ 3467.156659] Lustre: Skipped 25 previous similar messages [ 3467.282885] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 3467.305560] [] kthread+0xcf/0xe0 [ 3467.323493] [] ? kthread+0x0/0xe0 [ 3467.341460] [] ret_from_fork+0x58/0x90 [ 3467.360747] [] ? kthread+0x0/0xe0 [ 3467.378501] [ 3467.384716] LustreError: dumping log to /tmp/lustre-log.1513213587.139711 [ 3467.460834] Pid: 139710, comm: ll_ost_io01_115 [ 3467.476951] Call Trace: [ 3467.493746] [] schedule+0x29/0x70 [ 3467.511801] [] cv_wait_common+0x125/0x150 [spl] [ 3467.533715] [] ? autoremove_wake_function+0x0/0x40 [ 3467.556363] [] __cv_wait+0x15/0x20 [spl] [ 3467.576073] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 3467.596773] [] dmu_tx_assign+0x91/0x490 [zfs] [ 3467.617666] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 3467.640199] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 3467.661252] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 3467.683958] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 3467.704700] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 3467.726218] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 3467.748219] [] ? task_rq_unlock+0x1a/0x20 [ 3467.767607] [] ? __sched_setscheduler+0x882/0x8c0 [ 3467.789289] [] ? ___slab_alloc+0x209/0x4f0 [ 3467.801773] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334722 [ 3467.841760] [] ? __enqueue_entity+0x78/0x80 [ 3467.862345] [] ? enqueue_entity+0x26c/0xb60 [ 3467.882912] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 3467.906545] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 3467.930400] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 3467.956775] [] ? ptlrpc_start_thread+0x26d/0xbd0 [ptlrpc] [ 3467.981047] [] ? __wake_up_common+0x58/0x90 [ 3468.001303] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 3468.022905] [] ? __switch_to+0xd7/0x510 [ 3468.041898] [] ? __schedule+0x370/0x8b0 [ 3468.060871] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 3468.082390] [] kthread+0xcf/0xe0 [ 3468.099264] [] ? kthread+0x0/0xe0 [ 3468.116363] [] ret_from_fork+0x58/0x90 [ 3468.134893] [] ? kthread+0x0/0xe0 [ 3468.151977] [ 3468.157603] Pid: 139709, comm: ll_ost_io01_114 [ 3468.172968] Call Trace: [ 3468.187283] [] schedule+0x29/0x70 [ 3468.204325] [] cv_wait_common+0x125/0x150 [spl] [ 3468.225349] [] ? autoremove_wake_function+0x0/0x40 [ 3468.247216] [] __cv_wait+0x15/0x20 [spl] [ 3468.266256] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 3468.286430] [] dmu_tx_assign+0x91/0x490 [zfs] [ 3468.306876] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 3468.329052] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 3468.349789] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 3468.372287] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 3468.392757] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 3468.414077] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 3468.435920] [] ? task_rq_unlock+0x1a/0x20 [ 3468.455201] [] ? __sched_setscheduler+0x882/0x8c0 [ 3468.476779] [] ? __enqueue_entity+0x78/0x80 [ 3468.485391] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629793 [ 3468.529949] [] ? enqueue_entity+0x26c/0xb60 [ 3468.550565] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 3468.574232] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 3468.598138] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 3468.624550] [] ? ptlrpc_start_thread+0x26d/0xbd0 [ptlrpc] [ 3468.648888] [] ? __wake_up_common+0x58/0x90 [ 3468.669201] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 3468.690859] [] ? __switch_to+0xd7/0x510 [ 3468.709872] [] ? __schedule+0x371/0x8b0 [ 3468.728887] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 3468.750363] [] kthread+0xcf/0xe0 [ 3468.767230] [] ? kthread+0x0/0xe0 [ 3468.784328] [] ret_from_fork+0x58/0x90 [ 3468.802835] [] ? kthread+0x0/0xe0 [ 3468.819861] [ 3468.825437] Pid: 139708, comm: ll_ost_io01_113 [ 3468.840732] Call Trace: [ 3468.854949] [] schedule+0x29/0x70 [ 3468.871924] [] cv_wait_common+0x125/0x150 [spl] [ 3468.892944] [] ? autoremove_wake_function+0x0/0x40 [ 3468.914815] [] __cv_wait+0x15/0x20 [spl] [ 3468.933844] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 3468.954012] [] dmu_tx_assign+0x91/0x490 [zfs] [ 3468.974438] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 3468.996591] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 3469.017323] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 3469.039770] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 3469.060241] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 3469.081559] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 3469.103421] [] ? task_rq_unlock+0x1a/0x20 [ 3469.122716] [] ? __sched_setscheduler+0x882/0x8c0 [ 3469.144306] [] ? sched_set_stop_task+0x80/0xb0 [ 3469.165060] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 3469.188100] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 3469.211431] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 3469.237353] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 3469.261267] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 3469.282569] [] ? __switch_to+0xd7/0x510 [ 3469.301307] [] ? __schedule+0x39d/0x8b0 [ 3469.320052] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 3469.341362] [] kthread+0xcf/0xe0 [ 3469.358053] [] ? kthread+0x0/0xe0 [ 3469.374996] [] ret_from_fork+0x58/0x90 [ 3469.393365] [] ? kthread+0x0/0xe0 [ 3469.410268] [ 3469.415716] Pid: 139707, comm: ll_ost_io01_112 [ 3469.430913] Call Trace: [ 3469.444903] [] schedule+0x29/0x70 [ 3469.461766] [] cv_wait_common+0x125/0x150 [spl] [ 3469.482628] [] ? autoremove_wake_function+0x0/0x40 [ 3469.504344] [] __cv_wait+0x15/0x20 [spl] [ 3469.506247] Lustre: soaked-OST0005: deleting orphan objects from 0x0:24614878 to 0x0:24626244 [ 3469.551857] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 3469.571903] [] dmu_tx_assign+0x91/0x490 [zfs] [ 3469.592204] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 3469.614255] [] ofd_trans_start+0x6b/0xe0 [ofd] [ 3469.634871] [] ofd_commitrw_write+0x94b/0x1c50 [ofd] [ 3469.657219] [] ofd_commitrw+0x4b9/0xac0 [ofd] [ 3469.677575] [] obd_commitrw+0x2ed/0x330 [ptlrpc] [ 3469.698786] [] tgt_brw_write+0xff1/0x17c0 [ptlrpc] [ 3469.720544] [] ? task_rq_unlock+0x1a/0x20 [ 3469.739717] [] ? __sched_setscheduler+0x882/0x8c0 [ 3469.761194] [] ? sched_set_stop_task+0x80/0xb0 [ 3469.781831] [] ? target_bulk_timeout+0x0/0xb0 [ptlrpc] [ 3469.804767] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [ 3469.827984] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 3469.853794] [] ? ptlrpc_start_thread+0x74c/0xbd0 [ptlrpc] [ 3469.877579] [] ? kmalloc_order_trace+0x2e/0xa0 [ 3469.898241] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 3469.919442] [] ? __switch_to+0xd7/0x510 [ 3469.938076] [] ? __schedule+0x39d/0x8b0 [ 3469.956741] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 3469.977944] [] kthread+0xcf/0xe0 [ 3469.994574] [] ? kthread+0x0/0xe0 [ 3470.011490] [] ret_from_fork+0x58/0x90 [ 3470.029831] [] ? kthread+0x0/0xe0 [ 3470.046734] [ 3470.052194] LNet: Service thread pid 139706 was inactive for 1204.83s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [ 3470.096360] LNet: Skipped 53 previous similar messages [ 3470.643748] LustreError: dumping log to /tmp/lustre-log.1513213590.136864 [ 3472.274735] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052242 [ 3472.485730] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10621985 [ 3473.485462] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608289 [ 3473.485934] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610209 [ 3475.259566] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036337 [ 3476.097678] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273665 [ 3478.242309] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028225 [ 3478.242668] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032434 [ 4002.042676] Lustre: 5705:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1513214115/real 0] req@ffff8803e5aa2100 x1586716053515568/t0(0) o400->soaked-MDT0000-lwp-OST0005@192.168.1.108@o2ib:12/10 lens 224/224 e 0 to 1 dl 1513214122 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4002.042682] Lustre: 5708:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1513214115/real 0] req@ffff8803e5aa1200 x1586716053515632/t0(0) o400->soaked-MDT0000-lwp-OST000b@192.168.1.108@o2ib:12/10 lens 224/224 e 0 to 1 dl 1513214122 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4002.042688] Lustre: soaked-MDT0000-lwp-OST0011: Connection to soaked-MDT0000 (at 192.168.1.108@o2ib) was lost; in progress operations using this service will wait for recovery to complete [ 4002.042691] Lustre: 5708:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4002.042697] LustreError: 166-1: MGC192.168.1.108@o2ib: Connection to MGS (at 192.168.1.108@o2ib) was lost; in progress operations using this service will fail [ 4008.170502] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1513214122/real 0] req@ffff8803e5aa0f00 x1586716053515824/t0(0) o38->soaked-MDT0000-lwp-OST0005@192.168.1.108@o2ib:12/10 lens 520/544 e 0 to 1 dl 1513214128 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4033.170098] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513214147/real 1513214147] req@ffff8803e5aa2d00 x1586716053515840/t0(0) o250->MGC192.168.1.108@o2ib@192.168.1.109@o2ib:26/25 lens 520/544 e 0 to 1 dl 1513214153 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4033.209030] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4052.170407] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1513214172/real 1513214172] req@ffff8803e5aa3300 x1586716053516320/t0(0) o38->soaked-MDT0000-lwp-OST0011@192.168.1.108@o2ib:12/10 lens 520/544 e 0 to 1 dl 1513214183 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [ 4062.505610] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807d75c6850 x1586704152849504/t0(0) o4->2ceed2c4-2b59-dbaa-1c06-3e6d253475aa@192.168.1.137@o2ib:152/0 lens 608/0 e 0 to 0 dl 1513214187 ref 2 fl New:/2/ffffffff rc 0/-1 [ 4062.549333] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 298 previous similar messages [ 4068.141764] Lustre: soaked-OST0005: Client 8a80e4dc-a534-30af-d839-075339342992 (at 192.168.1.130@o2ib) reconnecting [ 4068.156354] Lustre: Skipped 25 previous similar messages [ 4068.162598] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 4068.162601] Lustre: Skipped 25 previous similar messages [ 4068.743641] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334754 [ 4069.419039] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629825 [ 4073.232318] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052274 [ 4073.419472] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622017 [ 4074.419371] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610241 [ 4074.419803] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608321 [ 4076.233134] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036369 [ 4077.086737] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273697 [ 4079.233140] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028257 [ 4079.233780] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032466 [ 4088.169203] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513214197/real 1513214197] req@ffff8803e5aa6f00 x1586716053516384/t0(0) o250->MGC192.168.1.108@o2ib@192.168.1.109@o2ib:26/25 lens 520/544 e 0 to 1 dl 1513214208 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4088.209871] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4102.169289] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1513214222/real 1513214222] req@ffff8803fb7c4500 x1586716053516800/t0(0) o38->soaked-MDT0000-lwp-OST000b@192.168.1.108@o2ib:12/10 lens 520/544 e 0 to 1 dl 1513214238 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 [ 4143.168241] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513214247/real 1513214247] req@ffff8803e5aa5700 x1586716053516928/t0(0) o250->MGC192.168.1.108@o2ib@192.168.1.109@o2ib:26/25 lens 520/544 e 0 to 1 dl 1513214263 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4143.209628] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [ 4198.167297] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513214297/real 1513214297] req@ffff8803e5aa3000 x1586716053517472/t0(0) o250->MGC192.168.1.108@o2ib@192.168.1.109@o2ib:26/25 lens 520/544 e 0 to 1 dl 1513214318 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4198.208542] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [ 4217.240267] Lustre: soaked-OST000b: haven't heard from client soaked-MDT0000-mdtlov_UUID (at 192.168.1.108@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880820423400, cur 1513214337 expire 1513214187 last 1513214110 [ 4283.165832] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513214372/real 1513214387] req@ffff8803d6742700 x1586716053518480/t0(0) o38->soaked-MDT0000-lwp-OST0011@192.168.1.108@o2ib:12/10 lens 520/544 e 0 to 1 dl 1513214403 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [ 4283.269951] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [ 4314.540149] Lustre: soaked-OST0011: Received new MDS connection from 192.168.1.108@o2ib, removing former export from same NID [ 4327.165754] LustreError: 167-0: soaked-MDT0000-lwp-OST0017: This client was evicted by soaked-MDT0000; in progress operations using this service will fail. [ 4362.366778] LustreError: 11-0: soaked-MDT0000-lwp-OST0011: operation lfsck_notify to node 192.168.1.108@o2ib failed: rc = -11 [ 4363.346461] LustreError: 11-0: soaked-MDT0000-lwp-OST000b: operation lfsck_notify to node 192.168.1.108@o2ib failed: rc = -11 [ 4363.386568] LustreError: Skipped 1 previous similar message [ 4364.586101] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4376.808801] LustreError: 11-0: soaked-MDT0000-lwp-OST0017: operation lfsck_notify to node 192.168.1.108@o2ib failed: rc = -11 [ 4376.849120] LustreError: Skipped 1 previous similar message [ 4377.164771] Lustre: Evicted from MGS (at MGC192.168.1.108@o2ib_0) after server handle changed from 0x7c5925bf036f3a4d to 0x472904aca8d4e15f [ 4414.585261] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4464.584304] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4514.583481] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4515.105806] LNet: Service thread pid 9102 was inactive for 200.65s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [ 4515.167391] LNet: Skipped 4 previous similar messages [ 4515.186945] Pid: 9102, comm: ll_ost01_012 [ 4515.203001] Call Trace: [ 4515.220877] [] schedule+0x29/0x70 [ 4515.239561] [] cv_wait_common+0x125/0x150 [spl] [ 4515.262135] [] ? autoremove_wake_function+0x0/0x40 [ 4515.285433] [] __cv_wait+0x15/0x20 [spl] [ 4515.305792] [] dmu_tx_wait+0x20b/0x3c0 [zfs] [ 4515.327209] [] dmu_tx_assign+0x91/0x490 [zfs] [ 4515.348773] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [ 4515.372054] [] tgt_client_data_update+0x2ed/0x5d0 [ptlrpc] [ 4515.397175] [] tgt_client_new+0x41b/0x610 [ptlrpc] [ 4515.419904] [] ofd_obd_connect+0x3a3/0x4c0 [ofd] [ 4515.441968] [] target_handle_connect+0x12bc/0x3200 [ptlrpc] [ 4515.467102] [] ? enqueue_task_fair+0x208/0x6c0 [ 4515.488426] [] ? sched_clock_cpu+0x85/0xc0 [ 4515.508583] [] tgt_request_handle+0x402/0x1370 [ptlrpc] [ 4515.532402] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [ 4515.558792] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [ 4515.582274] [] ? default_wake_function+0x12/0x20 [ 4515.604002] [] ? __wake_up_common+0x58/0x90 [ 4515.624329] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [ 4515.646076] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [ 4515.667785] [] kthread+0xcf/0xe0 [ 4515.684869] [] ? kthread+0x0/0xe0 [ 4515.702220] [] ret_from_fork+0x58/0x90 [ 4515.720986] [] ? kthread+0x0/0xe0 [ 4515.738276] [ 4515.744115] LustreError: dumping log to /tmp/lustre-log.1513214635.9102 [ 4564.582549] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4600.332345] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [ 4614.581726] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4664.580864] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4664.607251] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807d8729050 x1586704163825504/t0(0) o4->627621b7-cb99-7763-2256-7aec3c9afc32@192.168.1.138@o2ib:754/0 lens 608/0 e 0 to 0 dl 1513214789 ref 2 fl New:/2/ffffffff rc 0/-1 [ 4664.607255] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 380 previous similar messages [ 4669.152061] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 4669.188786] Lustre: Skipped 23 previous similar messages [ 4669.208458] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 4669.245908] Lustre: Skipped 32 previous similar messages [ 4669.732368] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334786 [ 4670.432320] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629857 [ 4674.318459] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052306 [ 4674.432868] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622049 [ 4675.434013] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610273 [ 4675.434234] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608353 [ 4677.221463] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036401 [ 4678.075915] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273729 [ 4680.222604] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028289 [ 4680.223188] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032498 [ 4714.579968] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4814.578102] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4814.691718] Lustre: Skipped 1 previous similar message [ 4964.575547] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 4964.608445] Lustre: Skipped 2 previous similar messages [ 5264.570101] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 5264.660937] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8805dc18b450 x1586704091559456/t0(0) o4->8a80e4dc-a534-30af-d839-075339342992@192.168.1.130@o2ib:599/0 lens 608/0 e 0 to 0 dl 1513215389 ref 2 fl New:/2/ffffffff rc 0/-1 [ 5264.660941] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 381 previous similar messages [ 5265.162379] Lustre: Skipped 5 previous similar messages [ 5270.141523] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 5270.250016] Lustre: soaked-OST0005: Connection restored to 13292baa-da7b-6c1c-3d65-2f4969d8fa28 (at 192.168.1.134@o2ib) [ 5270.250019] Lustre: Skipped 23 previous similar messages [ 5270.467634] Lustre: Skipped 27 previous similar messages [ 5270.673690] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334818 [ 5271.446336] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629889 [ 5275.347798] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052338 [ 5275.446520] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622081 [ 5276.447202] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610305 [ 5276.447634] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608385 [ 5278.212110] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036433 [ 5279.064928] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273761 [ 5281.212765] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028321 [ 5281.213613] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032530 [ 5332.319773] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [ 5481.783455] Lustre: soaked-OST0011: deleting orphan objects from 0x0:24610341 to 0x0:24621761 [ 5481.783476] Lustre: soaked-OST0017: deleting orphan objects from 0x0:24555996 to 0x0:24568214 [ 5481.783478] Lustre: soaked-OST000b: deleting orphan objects from 0x0:24650207 to 0x0:24661573 [ 5698.313490] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [ 5814.560545] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 5814.674475] Lustre: Skipped 10 previous similar messages [ 5866.762646] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff88068d41dc50 x1586709579097600/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:445/0 lens 520/0 e 0 to 0 dl 1513215990 ref 2 fl New:/2/ffffffff rc 0/-1 [ 5867.109066] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 499 previous similar messages [ 5871.131079] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 5871.241720] Lustre: soaked-OST0005: Connection restored to 13292baa-da7b-6c1c-3d65-2f4969d8fa28 (at 192.168.1.134@o2ib) [ 5871.241723] Lustre: Skipped 24 previous similar messages [ 5871.457454] Lustre: Skipped 24 previous similar messages [ 5871.872706] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334850 [ 5872.379847] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629921 [ 5876.380124] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622113 [ 5876.672793] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052370 [ 5877.379940] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610337 [ 5877.380137] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608417 [ 5879.203146] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036465 [ 5880.054033] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273793 [ 5882.203700] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028353 [ 5882.204438] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032562 [ 6064.307200] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [ 6430.300922] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 2 seconds [ 6464.548841] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 6464.561597] Lustre: Skipped 12 previous similar messages [ 6466.816321] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8806ec758c50 x1586704152852048/t0(0) o4->2ceed2c4-2b59-dbaa-1c06-3e6d253475aa@192.168.1.137@o2ib:291/0 lens 608/0 e 0 to 0 dl 1513216591 ref 2 fl New:/2/ffffffff rc 0/-1 [ 6466.857051] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 263 previous similar messages [ 6472.120424] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 6472.134534] Lustre: Skipped 21 previous similar messages [ 6472.142899] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 6472.157340] Lustre: Skipped 23 previous similar messages [ 6472.811679] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334882 [ 6473.409198] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629953 [ 6477.409838] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622145 [ 6477.588019] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052402 [ 6478.409659] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608449 [ 6478.410034] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610369 [ 6480.192850] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036497 [ 6481.043102] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273825 [ 6483.194129] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028385 [ 6483.194773] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032594 [ 7068.917945] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8807c89b2450 x1586709579103936/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:137/0 lens 520/0 e 0 to 0 dl 1513217192 ref 2 fl New:/2/ffffffff rc 0/-1 [ 7068.959695] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 454 previous similar messages [ 7073.109957] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 7073.124496] Lustre: Skipped 24 previous similar messages [ 7073.133030] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 7073.147630] Lustre: Skipped 24 previous similar messages [ 7073.763069] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334914 [ 7074.358570] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10629985 [ 7078.359596] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622177 [ 7078.625157] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052434 [ 7079.359140] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608481 [ 7079.359533] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610401 [ 7081.183670] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036529 [ 7082.032222] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273857 [ 7084.184559] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028417 [ 7084.185129] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032626 [ 7114.537564] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 7114.550630] Lustre: Skipped 12 previous similar messages [ 7466.283124] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [ 7529.282034] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [ 7668.971665] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (4/4), not sending early reply req@ffff8807c89f2c50 x1586709579108272/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:738/0 lens 520/0 e 0 to 0 dl 1513217793 ref 2 fl New:/2/ffffffff rc 0/-1 [ 7669.319041] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 424 previous similar messages [ 7673.855496] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 7673.984272] Lustre: Skipped 24 previous similar messages [ 7674.050555] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 7674.182244] Lustre: Skipped 24 previous similar messages [ 7674.653739] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334946 [ 7675.373614] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10630017 [ 7679.355855] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052466 [ 7679.372723] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622209 [ 7680.374152] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610433 [ 7680.374459] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608513 [ 7682.174228] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036561 [ 7683.021503] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273889 [ 7685.174964] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032658 [ 7685.175562] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028449 [ 7742.545548] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 7742.659436] Lustre: Skipped 11 previous similar messages [ 8261.269493] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [ 8271.073321] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (3/3), not sending early reply req@ffff8805d97a9c50 x1586709579120832/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:584/0 lens 520/0 e 0 to 0 dl 1513218394 ref 2 fl New:/2/ffffffff rc 0/-1 [ 8271.073325] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880785f1ac50 x1586704163831296/t0(0) o4->627621b7-cb99-7763-2256-7aec3c9afc32@192.168.1.138@o2ib:586/0 lens 608/0 e 0 to 0 dl 1513218396 ref 2 fl New:/2/ffffffff rc 0/-1 [ 8271.073332] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 264 previous similar messages [ 8271.317598] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 116 previous similar messages [ 8274.844991] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 8274.882212] Lustre: Skipped 24 previous similar messages [ 8274.902421] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 8274.940372] Lustre: Skipped 22 previous similar messages [ 8275.731249] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9334978 [ 8276.482051] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10630049 [ 8280.304971] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052498 [ 8280.482460] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622241 [ 8281.484118] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608545 [ 8281.484863] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610465 [ 8283.164507] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036593 [ 8284.010407] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273921 [ 8286.165477] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028481 [ 8286.166249] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032690 [ 8392.534039] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 8392.648215] Lustre: Skipped 12 previous similar messages [ 8627.263193] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [ 8871.127018] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880752cc2850 x1586709579105392/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:431/0 lens 520/0 e 0 to 0 dl 1513218996 ref 2 fl New:/2/ffffffff rc 0/-1 [ 8871.127022] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807ecee0c50 x1586704091559360/t0(0) o4->8a80e4dc-a534-30af-d839-075339342992@192.168.1.130@o2ib:431/0 lens 608/0 e 0 to 0 dl 1513218996 ref 2 fl New:H/2/ffffffff rc 0/-1 [ 8871.127029] Lustre: 139712:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 263 previous similar messages [ 8871.956404] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 69 previous similar messages [ 8875.834366] Lustre: soaked-OST0005: Client 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) reconnecting [ 8875.963393] Lustre: Skipped 24 previous similar messages [ 8876.029879] Lustre: soaked-OST0005: Connection restored to 2ceed2c4-2b59-dbaa-1c06-3e6d253475aa (at 192.168.1.137@o2ib) [ 8876.029881] Lustre: Skipped 24 previous similar messages [ 8876.680161] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9333980 to 0x440000402:9335010 [ 8877.416863] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10628904 to 0x440000401:10630081 [ 8881.417047] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10621488 to 0x5c0000402:10622273 [ 8881.906667] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9051958 to 0x440000400:9052530 [ 8882.418295] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10607610 to 0x8c0000401:10608577 [ 8882.418706] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10609907 to 0x740000402:10610497 [ 8884.155231] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036054 to 0x5c0000401:9036625 [ 8884.999645] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9273377 to 0x740000401:9273953 [ 8887.155896] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9027942 to 0x740000400:9028513 [ 8887.156526] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032183 to 0x8c0000400:9032722 [ 8993.256891] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [ 8993.271029] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [ 9042.522585] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 9042.535805] Lustre: Skipped 12 previous similar messages [ 9359.250611] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [ 9359.375203] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [ 9471.180721] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8807c7eb2c50 x1586709579103936/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:276/0 lens 520/0 e 0 to 0 dl 1513219596 ref 2 fl New:/2/ffffffff rc 0/-1 [ 9471.222560] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 309 previous similar messages [ 9477.467076] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [ 9477.477421] Lustre: soaked-OST0005: Connection restored to 13292baa-da7b-6c1c-3d65-2f4969d8fa28 (at 192.168.1.134@o2ib) [ 9477.477424] Lustre: Skipped 24 previous similar messages [ 9477.504849] Lustre: Skipped 25 previous similar messages [ 9477.629207] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336034 [ 9478.365355] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630689 [ 9482.365725] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10622913 [ 9482.402071] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052562 [ 9483.366564] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611233 [ 9483.366881] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609313 [ 9485.145838] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036657 [ 9485.988736] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9274977 [ 9488.146421] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028545 [ 9488.147219] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032754 [ 9692.511057] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [ 9692.524226] Lustre: Skipped 12 previous similar messages [10072.642383] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880344bbe850 x1586704105275408/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:122/0 lens 608/0 e 0 to 0 dl 1513220197 ref 2 fl New:/2/ffffffff rc 0/-1 [10072.684452] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 678 previous similar messages [10078.308820] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [10078.323194] Lustre: Skipped 64 previous similar messages [10078.331599] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [10078.346102] Lustre: Skipped 65 previous similar messages [10078.586303] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336066 [10079.322683] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630721 [10083.323050] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10622945 [10083.351343] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052594 [10084.324319] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611265 [10084.324633] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609345 [10086.135914] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036689 [10086.977819] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9275009 [10089.137206] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028577 [10089.137963] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032786 [10342.499490] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [10342.532815] Lustre: Skipped 12 previous similar messages [10395.232822] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [10395.268897] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [10673.656065] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880362996850 x1586704105275408/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:723/0 lens 608/0 e 0 to 0 dl 1513220798 ref 2 fl New:/2/ffffffff rc 0/-1 [10674.011473] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 725 previous similar messages [10679.164972] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [10679.293432] Lustre: Skipped 63 previous similar messages [10679.359465] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [10679.491021] Lustre: Skipped 64 previous similar messages [10679.679500] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336098 [10680.384451] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630753 [10684.269033] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052626 [10684.384692] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10622977 [10685.385561] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609377 [10685.385951] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611297 [10687.126649] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036721 [10687.966856] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9275041 [10690.127474] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028609 [10690.128400] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032818 [10992.488018] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [10992.501190] Lustre: Skipped 12 previous similar messages [11127.220240] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [11274.797739] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803cfd56c50 x1586704105278368/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:569/0 lens 608/0 e 0 to 0 dl 1513221399 ref 2 fl New:/2/ffffffff rc 0/-1 [11274.903036] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 704 previous similar messages [11280.154410] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [11280.191609] Lustre: Skipped 63 previous similar messages [11280.211658] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [11280.249516] Lustre: Skipped 64 previous similar messages [11280.708633] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336130 [11281.327237] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630785 [11285.258408] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052658 [11285.326402] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623009 [11286.327234] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611329 [11286.327561] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609409 [11288.117075] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036753 [11288.955909] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9275073 [11291.117864] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028641 [11291.118173] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032850 [11493.213957] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [11642.476495] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [11642.489677] Lustre: Skipped 12 previous similar messages [11859.207666] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [11875.491438] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8806bf55a450 x1586709579109280/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:415/0 lens 520/0 e 0 to 0 dl 1513222000 ref 2 fl New:/2/ffffffff rc 0/-1 [11875.533641] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 704 previous similar messages [11881.054353] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [11881.068829] Lustre: Skipped 65 previous similar messages [11881.077410] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [11881.092072] Lustre: Skipped 65 previous similar messages [11881.705609] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336162 [11882.275656] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630817 [11886.276020] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623041 [11886.860920] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052690 [11887.276915] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611361 [11887.277318] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609441 [11889.107680] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036785 [11889.945115] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9275105 [11892.108208] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028673 [11892.108876] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032882 [12225.201394] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [12292.464995] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [12292.478262] Lustre: Skipped 12 previous similar messages [12475.929116] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88026b21b850 x1586704105272848/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:261/0 lens 608/0 e 0 to 0 dl 1513222601 ref 2 fl New:/2/ffffffff rc 0/-1 [12475.971882] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 704 previous similar messages [12482.043441] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [12482.057906] Lustre: Skipped 65 previous similar messages [12482.066466] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [12482.081130] Lustre: Skipped 65 previous similar messages [12482.638660] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336194 [12483.314249] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630849 [12487.157484] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052722 [12487.313617] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623073 [12488.314605] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611393 [12488.314998] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609473 [12490.098053] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036817 [12490.934252] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9275137 [12493.098928] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028705 [12493.099384] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032914 [12591.195097] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [12942.453435] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [12942.466690] Lustre: Skipped 12 previous similar messages [12957.188816] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [13077.006792] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880060bfb450 x1586704105272848/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:107/0 lens 608/0 e 0 to 0 dl 1513223202 ref 2 fl New:/2/ffffffff rc 0/-1 [13077.112352] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 704 previous similar messages [13083.032745] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [13083.069993] Lustre: Skipped 64 previous similar messages [13083.090098] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [13083.128000] Lustre: Skipped 64 previous similar messages [13083.611898] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336226 [13084.255088] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630881 [13088.187041] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052754 [13088.255241] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623105 [13089.256350] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609505 [13089.256765] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611425 [13091.088269] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036849 [13091.923285] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9275169 [13094.089129] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028737 [13094.089857] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032946 [13592.442002] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [13592.455216] Lustre: Skipped 12 previous similar messages [13627.177292] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [13678.084453] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88038894cc50 x1586704105275408/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:708/0 lens 608/0 e 0 to 0 dl 1513223803 ref 2 fl New:/2/ffffffff rc 0/-1 [13678.127164] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 703 previous similar messages [13684.022148] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [13684.036530] Lustre: Skipped 65 previous similar messages [13684.045059] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [13684.059706] Lustre: Skipped 65 previous similar messages [13684.552926] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9335945 to 0x440000402:9336258 [13685.468572] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630913 [13689.064300] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9052535 to 0x440000400:9052786 [13689.468687] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623137 [13690.244598] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611457 [13690.245017] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609537 [13692.064863] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9036630 to 0x5c0000401:9036881 [13692.912470] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9274884 to 0x740000401:9275201 [13695.064874] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9028518 to 0x740000400:9028769 [13695.065400] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9032728 to 0x8c0000400:9032978 [13993.171021] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [14242.430461] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [14242.544737] Lustre: Skipped 12 previous similar messages [14279.162127] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803370df450 x1586704105272848/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:554/0 lens 608/0 e 0 to 0 dl 1513224404 ref 2 fl New:/2/ffffffff rc 0/-1 [14279.518063] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 704 previous similar messages [14284.819660] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [14284.938060] Lustre: Skipped 62 previous similar messages [14285.004281] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [14285.125565] Lustre: Skipped 66 previous similar messages [14285.192464] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336322 [14286.395007] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630945 [14290.085888] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9053991 to 0x440000400:9054226 [14290.394410] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623169 [14291.386006] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609569 [14291.387150] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611489 [14293.055420] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038102 to 0x5c0000401:9038129 [14293.147965] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275265 [14296.056300] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9029990 to 0x740000400:9030017 [14296.057041] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034197 to 0x8c0000400:9034226 [14880.175818] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88030e9dd850 x1586704105273744/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:400/0 lens 608/0 e 0 to 0 dl 1513225005 ref 2 fl New:/2/ffffffff rc 0/-1 [14880.531543] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 709 previous similar messages [14885.793114] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [14885.911546] Lustre: Skipped 67 previous similar messages [14885.978225] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [14886.099576] Lustre: Skipped 67 previous similar messages [14886.166536] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336354 [14887.327971] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10630977 [14891.043250] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054258 [14891.328051] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623201 [14892.328379] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611521 [14892.329260] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609601 [14892.418970] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [14892.418974] Lustre: Skipped 12 previous similar messages [14894.043789] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038161 [14894.137321] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275297 [14897.044007] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030049 [14897.044674] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034258 [15028.153248] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [15394.146959] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [15481.765486] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803b7655450 x1586704102713184/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:246/0 lens 608/0 e 0 to 0 dl 1513225606 ref 2 fl New:H/2/ffffffff rc 0/-1 [15481.870910] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 810 previous similar messages [15486.766483] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [15486.801087] Lustre: Skipped 78 previous similar messages [15486.821161] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [15486.856147] Lustre: Skipped 76 previous similar messages [15486.877021] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336386 [15488.261410] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631009 [15492.246895] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054290 [15492.261492] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623233 [15493.262589] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611553 [15493.262881] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609633 [15495.033431] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038193 [15495.126959] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275329 [15498.034585] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034290 [15498.035188] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030081 [15542.407557] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [15542.440595] Lustre: Skipped 12 previous similar messages [15760.140677] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [16081.883189] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8800905a1c50 x1586704102712208/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:92/0 lens 608/0 e 0 to 0 dl 1513226207 ref 2 fl New:/2/ffffffff rc 0/-1 [16081.925437] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 762 previous similar messages [16087.716149] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [16087.729674] Lustre: Skipped 79 previous similar messages [16087.738148] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [16087.751737] Lustre: Skipped 79 previous similar messages [16087.760760] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336418 [16089.210903] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631041 [16093.094374] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054322 [16093.211093] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623265 [16094.212223] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609665 [16094.212416] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611585 [16096.024082] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038225 [16096.116588] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275361 [16099.024660] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030113 [16099.025323] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034322 [16126.134384] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [16192.396073] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [16192.429388] Lustre: Skipped 12 previous similar messages [16682.960859] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88033f00f450 x1586704102712208/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:693/0 lens 608/0 e 0 to 0 dl 1513226808 ref 2 fl New:/2/ffffffff rc 0/-1 [16683.003338] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [16688.967626] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [16688.981963] Lustre: Skipped 78 previous similar messages [16688.990277] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [16689.004662] Lustre: Skipped 78 previous similar messages [16689.272520] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336450 [16690.202230] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631073 [16694.043774] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054354 [16694.202716] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623297 [16695.203795] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611617 [16695.204192] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609697 [16697.014387] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038257 [16697.106054] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275393 [16700.015203] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034354 [16700.015883] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030145 [16842.384628] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [16842.498767] Lustre: Skipped 12 previous similar messages [17162.116584] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [17162.152791] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [17284.038527] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880362097850 x1586704102710928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:539/0 lens 608/0 e 0 to 0 dl 1513227409 ref 2 fl New:/2/ffffffff rc 0/-1 [17284.394264] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [17289.956452] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [17289.970862] Lustre: Skipped 79 previous similar messages [17289.979373] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [17289.993967] Lustre: Skipped 79 previous similar messages [17290.192090] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336482 [17291.215151] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631105 [17295.049559] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054386 [17295.215542] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623329 [17296.216267] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611649 [17296.216625] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609729 [17298.004971] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038289 [17298.095496] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275425 [17301.005898] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030177 [17301.006536] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034386 [17492.373238] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [17492.386205] Lustre: Skipped 12 previous similar messages [17885.052217] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803c61f2450 x1586704102711920/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:385/0 lens 608/0 e 0 to 0 dl 1513228010 ref 2 fl New:/2/ffffffff rc 0/-1 [17885.407969] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages [17890.945152] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [17891.043517] Lustre: soaked-OST0005: Connection restored to 13292baa-da7b-6c1c-3d65-2f4969d8fa28 (at 192.168.1.134@o2ib) [17891.043519] Lustre: Skipped 79 previous similar messages [17891.271371] Lustre: Skipped 81 previous similar messages [17891.337700] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336514 [17892.283008] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631137 [17894.104025] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [17894.228057] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [17896.008487] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054418 [17896.228983] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623361 [17897.230083] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611681 [17897.230433] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609761 [17898.995309] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038321 [17899.085022] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275457 [17901.996203] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034418 [17901.996799] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030209 [18142.361663] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [18142.475784] Lustre: Skipped 12 previous similar messages [18260.097739] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [18260.111717] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [18486.129869] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803d88afc50 x1586704102711376/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:231/0 lens 608/0 e 0 to 0 dl 1513228611 ref 2 fl New:/2/ffffffff rc 0/-1 [18486.172073] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [18491.933801] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [18491.948037] Lustre: Skipped 76 previous similar messages [18491.956265] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [18491.970588] Lustre: Skipped 78 previous similar messages [18492.116738] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336546 [18493.170861] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631169 [18497.052285] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054450 [18497.169584] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623393 [18498.170682] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611713 [18498.171167] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609793 [18499.985927] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038353 [18500.074800] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275489 [18502.986703] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030241 [18502.987187] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034450 [18626.091448] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 2 seconds [18626.215856] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [18792.350202] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [18792.383553] Lustre: Skipped 12 previous similar messages [18929.086237] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [19087.207558] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803f5624c50 x1586704102711920/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:77/0 lens 608/0 e 0 to 0 dl 1513229212 ref 2 fl New:/2/ffffffff rc 0/-1 [19087.561830] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [19092.922518] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [19093.025656] Lustre: soaked-OST0005: Connection restored to 13292baa-da7b-6c1c-3d65-2f4969d8fa28 (at 192.168.1.134@o2ib) [19093.025659] Lustre: Skipped 79 previous similar messages [19093.136896] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336578 [19093.368174] Lustre: Skipped 80 previous similar messages [19094.250127] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631201 [19098.001952] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054482 [19098.250165] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623425 [19099.246317] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611745 [19099.246604] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609825 [19100.976187] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038385 [19101.064283] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275521 [19103.977031] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030273 [19103.977712] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034482 [19442.338779] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [19442.372406] Lustre: Skipped 12 previous similar messages [19661.073675] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [19688.285225] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880321aa6c50 x1586704102713184/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:678/0 lens 608/0 e 0 to 0 dl 1513229813 ref 2 fl New:H/2/ffffffff rc 0/-1 [19688.390934] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [19693.911227] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [19693.948451] Lustre: Skipped 77 previous similar messages [19693.968530] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [19694.006415] Lustre: Skipped 79 previous similar messages [19694.086332] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336610 [19695.180861] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631233 [19698.951358] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054514 [19699.180978] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623457 [19700.181858] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611777 [19700.182209] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609857 [19701.951569] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038417 [19702.053857] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275553 [19704.952580] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030305 [19704.953223] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034514 [20028.067362] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [20092.327414] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [20092.360879] Lustre: Skipped 12 previous similar messages [20289.362919] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880385d99450 x1586704102711920/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:524/0 lens 608/0 e 0 to 0 dl 1513230414 ref 2 fl New:/2/ffffffff rc 0/-1 [20289.468682] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages [20294.899938] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [20294.937201] Lustre: Skipped 79 previous similar messages [20294.957297] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [20294.995197] Lustre: Skipped 78 previous similar messages [20295.044093] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336642 [20296.138420] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631265 [20300.045873] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054546 [20300.138469] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623489 [20301.139378] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611809 [20301.139753] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609889 [20302.940515] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038449 [20303.043633] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275585 [20305.941177] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030337 [20305.941945] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034546 [20394.061083] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [20742.315928] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [20742.349452] Lustre: Skipped 12 previous similar messages [20760.054801] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 3 seconds [20760.090872] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [20890.440568] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880090563c50 x1586704102712208/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:370/0 lens 608/0 e 0 to 0 dl 1513231015 ref 2 fl New:/2/ffffffff rc 0/-1 [20890.483179] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [20895.888594] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [20895.902959] Lustre: Skipped 75 previous similar messages [20895.911501] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [20895.926118] Lustre: Skipped 76 previous similar messages [20896.025641] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336674 [20897.152159] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631297 [20901.152210] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623521 [20901.410435] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054578 [20902.153070] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611841 [20902.153458] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609921 [20903.930719] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038481 [20904.033201] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275617 [20906.931670] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030369 [20906.932110] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034578 [21392.304411] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [21392.337869] Lustre: Skipped 12 previous similar messages [21491.518264] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880388ec8450 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:216/0 lens 608/0 e 0 to 0 dl 1513231616 ref 2 fl New:/2/ffffffff rc 0/-1 [21491.874175] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [21492.042224] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [21496.877300] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [21496.982221] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [21496.982224] Lustre: Skipped 77 previous similar messages [21496.983093] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336706 [21497.312602] Lustre: Skipped 79 previous similar messages [21498.232332] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631329 [21502.232727] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623553 [21502.391690] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054610 [21503.233614] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611873 [21503.234014] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609953 [21504.921347] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038513 [21505.022581] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275649 [21507.922179] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030401 [21507.922592] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034610 [21795.037020] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [22042.292969] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [22042.326578] Lustre: Skipped 12 previous similar messages [22092.595942] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803b7295050 x1586704102712208/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:62/0 lens 608/0 e 0 to 0 dl 1513232217 ref 2 fl New:/2/ffffffff rc 0/-1 [22092.701221] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages [22097.866034] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [22097.903322] Lustre: Skipped 76 previous similar messages [22097.923425] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [22097.924640] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336738 [22097.996057] Lustre: Skipped 80 previous similar messages [22099.171788] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631361 [22103.171421] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623585 [22103.276183] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054642 [22104.172456] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611905 [22104.172710] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10609985 [22105.911874] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038545 [22106.012258] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275681 [22108.912526] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034642 [22108.913064] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030433 [22161.030729] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [22527.024445] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [22527.038477] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [22692.281511] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [22692.395550] Lustre: Skipped 12 previous similar messages [22692.713647] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88033d208850 x1586704102712848/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:663/0 lens 608/0 e 0 to 0 dl 1513232818 ref 2 fl New:/2/ffffffff rc 0/-1 [22693.069172] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 794 previous similar messages [22698.854800] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [22698.873480] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [22698.873482] Lustre: Skipped 77 previous similar messages [22698.874295] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336770 [22699.290002] Lustre: Skipped 80 previous similar messages [22700.105056] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631393 [22704.104979] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623617 [22704.169101] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054674 [22705.106394] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611937 [22705.106693] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610017 [22706.902168] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038577 [22707.001792] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275713 [22709.903066] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030465 [22709.903604] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034674 [23196.012953] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [23196.027011] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [23293.791322] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803a3537050 x1586704102711376/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:509/0 lens 608/0 e 0 to 0 dl 1513233419 ref 2 fl New:/2/ffffffff rc 0/-1 [23293.896519] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 776 previous similar messages [23299.831052] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [23299.843445] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [23299.843448] Lustre: Skipped 79 previous similar messages [23299.923611] Lustre: Skipped 78 previous similar messages [23299.944175] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336802 [23301.339040] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631425 [23305.117810] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623649 [23305.120154] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054706 [23306.118689] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10611969 [23306.118924] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610049 [23307.892849] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038609 [23307.991410] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275745 [23310.893692] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030497 [23310.894356] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034706 [23342.270102] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [23342.303175] Lustre: Skipped 12 previous similar messages [23563.006662] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [23894.868989] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88058bac0050 x1586704102710544/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:355/0 lens 608/0 e 0 to 0 dl 1513234020 ref 2 fl New:/2/ffffffff rc 0/-1 [23894.911344] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [23900.772614] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [23900.786172] Lustre: Skipped 76 previous similar messages [23900.794679] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [23900.808276] Lustre: Skipped 79 previous similar messages [23900.817364] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336834 [23902.085087] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631457 [23906.067354] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623681 [23906.115806] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054738 [23907.068242] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612001 [23907.068454] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610081 [23908.883382] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038641 [23908.981001] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275777 [23911.884065] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034738 [23911.884728] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030529 [23929.000366] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [23992.258633] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [23992.291892] Lustre: Skipped 12 previous similar messages [24294.994089] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [24295.118505] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [24495.946688] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88038fa7e450 x1586704102712208/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:201/0 lens 608/0 e 0 to 0 dl 1513234621 ref 2 fl New:/2/ffffffff rc 0/-1 [24496.302120] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [24501.722145] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [24501.820926] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [24501.820930] Lustre: Skipped 78 previous similar messages [24502.038019] Lustre: Skipped 80 previous similar messages [24502.104532] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336866 [24503.105966] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631489 [24507.105935] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623713 [24507.349947] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054770 [24508.106182] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610113 [24508.106559] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612033 [24509.873742] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038673 [24509.970292] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275809 [24512.874568] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030561 [24512.875385] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034770 [24642.247192] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [24642.280554] Lustre: Skipped 12 previous similar messages [24660.987815] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 2 seconds [24963.982605] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [25096.960344] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803ab19a050 x1586704102712208/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:47/0 lens 608/0 e 0 to 0 dl 1513235222 ref 2 fl New:/2/ffffffff rc 0/-1 [25097.064761] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [25102.679716] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [25102.714119] Lustre: Skipped 76 previous similar messages [25102.734054] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [25102.768923] Lustre: Skipped 79 previous similar messages [25102.789674] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336898 [25104.047512] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631521 [25108.047545] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623745 [25108.096863] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054802 [25109.048501] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612065 [25109.048696] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610145 [25110.864320] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038705 [25110.960015] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275841 [25113.865047] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030593 [25113.865552] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034802 [25292.235770] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [25292.248831] Lustre: Skipped 12 previous similar messages [25329.976323] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [25695.970036] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [25698.038041] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803f44ff850 x1586704102712848/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:648/0 lens 608/0 e 0 to 0 dl 1513235823 ref 2 fl New:/2/ffffffff rc 0/-1 [25698.393063] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [25703.645211] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [25703.763406] Lustre: Skipped 79 previous similar messages [25703.798280] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [25703.798283] Lustre: Skipped 79 previous similar messages [25704.027285] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336930 [25705.129998] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631553 [25709.046814] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054834 [25709.130164] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623777 [25710.131093] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612097 [25710.131399] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610177 [25711.854893] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038737 [25711.949541] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275873 [25714.855466] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030625 [25714.856319] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034834 [25942.224238] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [25942.338007] Lustre: Skipped 12 previous similar messages [26061.963731] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [26299.115691] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803d7ebe850 x1586704102711376/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:494/0 lens 608/0 e 0 to 0 dl 1513236424 ref 2 fl New:/2/ffffffff rc 0/-1 [26299.157956] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [26304.570771] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [26304.584295] Lustre: Skipped 74 previous similar messages [26304.592766] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [26304.606341] Lustre: Skipped 78 previous similar messages [26304.615399] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336962 [26306.074671] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631585 [26309.987864] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054866 [26310.074870] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623809 [26311.075891] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612129 [26311.076202] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610209 [26312.845117] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038769 [26312.939155] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275905 [26315.846042] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030657 [26315.846752] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034866 [26427.957449] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [26592.212768] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [26592.225891] Lustre: Skipped 12 previous similar messages [26793.951169] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [26900.193382] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8802fb43d050 x1586704102712208/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:340/0 lens 608/0 e 0 to 0 dl 1513237025 ref 2 fl New:/2/ffffffff rc 0/-1 [26900.297940] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [26905.544383] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [26905.578947] Lustre: Skipped 77 previous similar messages [26905.599027] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [26905.634083] Lustre: Skipped 78 previous similar messages [26905.654910] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9336994 [26907.025233] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631617 [26911.024481] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623841 [26911.655409] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054898 [26912.025466] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612161 [26912.025749] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610241 [26913.835726] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038801 [26913.928834] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275937 [26916.836592] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030689 [26916.837262] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034898 [27242.201384] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [27242.214364] Lustre: Skipped 12 previous similar messages [27463.939658] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [27463.953695] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [27501.271055] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803b5aa0050 x1586704102710928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:186/0 lens 608/0 e 0 to 0 dl 1513237626 ref 2 fl New:/2/ffffffff rc 0/-1 [27501.313431] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [27506.518021] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [27506.531552] Lustre: Skipped 78 previous similar messages [27506.540035] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [27506.553631] Lustre: Skipped 78 previous similar messages [27506.562753] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337026 [27508.054728] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631649 [27511.894999] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054930 [27512.054128] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623873 [27513.054891] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610273 [27513.055162] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612193 [27514.826395] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038833 [27514.918472] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9275969 [27517.827144] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030721 [27517.827650] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034930 [27892.190008] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [27892.203173] Lustre: Skipped 12 previous similar messages [28102.348722] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88028fbdbc50 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:32/0 lens 608/0 e 0 to 0 dl 1513238227 ref 2 fl New:/2/ffffffff rc 0/-1 [28102.390979] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [28107.651672] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [28107.665026] Lustre: Skipped 79 previous similar messages [28107.673500] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [28107.687056] Lustre: Skipped 79 previous similar messages [28107.696143] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337058 [28109.004012] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631681 [28112.844398] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054962 [28113.003493] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623905 [28114.004680] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612225 [28114.005104] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610305 [28115.816340] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038865 [28115.907928] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276001 [28118.816576] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030753 [28118.817086] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034962 [28542.178418] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [28542.191462] Lustre: Skipped 12 previous similar messages [28561.920815] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [28561.934642] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [28703.426396] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88035b737850 x1586704102713184/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:633/0 lens 608/0 e 0 to 0 dl 1513238828 ref 2 fl New:H/2/ffffffff rc 0/-1 [28703.782954] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 794 previous similar messages [28708.641324] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [28708.741844] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [28708.741846] Lustre: Skipped 78 previous similar messages [28708.957279] Lustre: Skipped 79 previous similar messages [28709.024164] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337090 [28710.081445] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631713 [28713.873955] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9054994 [28714.081444] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623937 [28715.082387] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612257 [28715.082698] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610337 [28716.793188] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038897 [28716.897552] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276033 [28719.793855] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030785 [28719.794427] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9034994 [29192.167087] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [29192.180082] Lustre: Skipped 12 previous similar messages [29303.672069] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88060f885c50 x1586709579109616/t0(0) o1000->soaked-MDT0003-mdtlov_UUID@192.168.1.111@o2ib:479/0 lens 520/0 e 0 to 0 dl 1513239429 ref 2 fl New:/2/ffffffff rc 0/-1 [29303.774762] Lustre: 32074:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 776 previous similar messages [29309.558866] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [29309.593361] Lustre: Skipped 76 previous similar messages [29309.613351] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [29309.648275] Lustre: Skipped 77 previous similar messages [29309.668958] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337122 [29311.014836] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631745 [29314.831513] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055026 [29315.014948] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10623969 [29316.015929] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612289 [29316.016230] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610369 [29317.783657] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038929 [29317.887090] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276065 [29320.784498] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030817 [29320.785268] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035026 [29842.155624] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [29842.168611] Lustre: Skipped 12 previous similar messages [29904.621796] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88040437c450 x1586704102710928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:325/0 lens 608/0 e 0 to 0 dl 1513240030 ref 2 fl New:/2/ffffffff rc 0/-1 [29904.725926] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 783 previous similar messages [29910.557201] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [29910.591743] Lustre: Skipped 78 previous similar messages [29910.611791] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [29910.646734] Lustre: Skipped 78 previous similar messages [29910.667399] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337154 [29912.062175] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631777 [29915.781041] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055058 [29916.061685] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624001 [29917.062551] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612321 [29917.062804] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610401 [29918.774143] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038961 [29918.876751] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276097 [29921.775110] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035058 [29921.775631] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030849 [30329.890455] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 2 seconds [30492.144285] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [30492.157360] Lustre: Skipped 12 previous similar messages [30505.699500] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88037a9a7450 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:171/0 lens 608/0 e 0 to 0 dl 1513240631 ref 2 fl New:/2/ffffffff rc 0/-1 [30505.741726] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [30511.578887] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [30511.591861] Lustre: Skipped 77 previous similar messages [30511.599799] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [30511.612578] Lustre: Skipped 78 previous similar messages [30511.620848] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337186 [30512.994281] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631809 [30516.986534] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055090 [30516.994405] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624033 [30517.995164] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610433 [30517.995471] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612353 [30519.764679] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9038993 [30519.866354] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276129 [30522.765727] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030881 [30522.766260] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035090 [31106.777121] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803f59a0450 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:17/0 lens 608/0 e 0 to 0 dl 1513241232 ref 2 fl New:/2/ffffffff rc 0/-1 [31106.819273] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [31112.519528] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [31112.533168] Lustre: Skipped 76 previous similar messages [31112.541755] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [31112.555436] Lustre: Skipped 77 previous similar messages [31112.564715] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337218 [31113.944434] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631841 [31117.943801] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624065 [31118.025405] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055122 [31118.944801] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612385 [31118.945102] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610465 [31120.755241] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039025 [31120.855901] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276161 [31123.756018] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030913 [31123.756458] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035122 [31142.133078] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [31142.146084] Lustre: Skipped 12 previous similar messages [31707.790787] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880077f89c50 x1586704102713184/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:618/0 lens 608/0 e 0 to 0 dl 1513241833 ref 2 fl New:H/2/ffffffff rc 0/-1 [31707.833254] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages [31713.501127] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [31713.514747] Lustre: Skipped 78 previous similar messages [31713.523310] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [31713.536972] Lustre: Skipped 77 previous similar messages [31713.548407] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337250 [31714.942230] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631873 [31718.941453] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624097 [31719.030488] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055154 [31719.942368] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610497 [31719.942715] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612417 [31721.745619] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039057 [31721.845446] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276193 [31724.746521] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030945 [31724.746931] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035154 [31792.121739] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [31792.154929] Lustre: Skipped 12 previous similar messages [32308.868469] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88042d28a450 x1586704102713168/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:464/0 lens 608/0 e 0 to 0 dl 1513242434 ref 2 fl New:H/2/ffffffff rc 0/-1 [32308.910548] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [32314.442740] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [32314.456342] Lustre: Skipped 79 previous similar messages [32314.464860] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [32314.478541] Lustre: Skipped 79 previous similar messages [32314.487805] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337282 [32315.964693] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631905 [32319.786786] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055186 [32319.964122] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624129 [32320.965074] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612449 [32320.965595] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610529 [32322.736367] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039089 [32322.835183] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276225 [32325.737013] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035186 [32325.737529] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9030977 [32442.110506] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [32442.143730] Lustre: Skipped 12 previous similar messages [32891.846439] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [32909.946165] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803e73e0450 x1586704102710928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:310/0 lens 608/0 e 0 to 0 dl 1513243035 ref 2 fl New:/2/ffffffff rc 0/-1 [32910.301620] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages [32915.489381] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [32915.607745] Lustre: Skipped 78 previous similar messages [32915.662882] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [32915.662885] Lustre: Skipped 79 previous similar messages [32915.872580] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337314 [32916.955138] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631937 [32920.848655] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055218 [32920.955071] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624161 [32921.955739] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612481 [32921.956093] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610561 [32923.726772] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039121 [32923.824682] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276257 [32926.727502] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035218 [32926.728213] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031009 [33092.099150] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [33092.132424] Lustre: Skipped 12 previous similar messages [33258.840124] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [33511.023830] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880345556450 x1586704102711376/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:156/0 lens 608/0 e 0 to 0 dl 1513243636 ref 2 fl New:/2/ffffffff rc 0/-1 [33511.129116] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [33516.421914] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [33516.456483] Lustre: Skipped 76 previous similar messages [33516.476550] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [33516.511528] Lustre: Skipped 78 previous similar messages [33516.532509] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337346 [33517.902569] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10631969 [33521.790019] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055250 [33521.902631] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624193 [33522.903442] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610593 [33522.903716] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612513 [33524.717324] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039153 [33524.814399] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276289 [33527.717991] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031041 [33527.718494] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035250 [33742.087977] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [33742.121313] Lustre: Skipped 12 previous similar messages [34112.101505] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88006f293c50 x1586704102711920/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:2/0 lens 608/0 e 0 to 0 dl 1513244237 ref 2 fl New:/2/ffffffff rc 0/-1 [34112.206243] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [34117.501075] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [34117.535635] Lustre: Skipped 78 previous similar messages [34117.555717] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [34117.590701] Lustre: Skipped 77 previous similar messages [34117.612743] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337378 [34118.908410] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632001 [34122.747488] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055282 [34122.908102] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624225 [34123.909025] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612545 [34123.909347] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610625 [34125.707684] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039185 [34125.803914] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276321 [34128.708516] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035282 [34128.709099] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031073 [34392.076757] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [34392.190827] Lustre: Skipped 12 previous similar messages [34659.816062] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [34713.179216] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803b78cf850 x1586704102710544/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:603/0 lens 608/0 e 0 to 0 dl 1513244838 ref 2 fl New:/2/ffffffff rc 0/-1 [34713.221706] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [34718.507919] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [34718.521518] Lustre: Skipped 77 previous similar messages [34718.530040] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [34718.543662] Lustre: Skipped 79 previous similar messages [34718.552917] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337410 [34719.881602] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632033 [34723.697136] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055314 [34723.881929] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624257 [34724.881725] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612577 [34724.882163] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610657 [34726.697060] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039217 [34726.793560] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276353 [34729.697154] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031105 [34729.697575] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035314 [35042.065502] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [35042.179581] Lustre: Skipped 12 previous similar messages [35314.256855] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880303003c50 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:449/0 lens 608/0 e 0 to 0 dl 1513245439 ref 2 fl New:/2/ffffffff rc 0/-1 [35314.298475] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [35319.438743] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [35319.452327] Lustre: Skipped 79 previous similar messages [35319.460864] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [35319.474471] Lustre: Skipped 79 previous similar messages [35319.483605] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337442 [35320.905403] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632065 [35324.742454] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055346 [35324.904354] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624289 [35325.905176] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612609 [35325.905486] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610689 [35327.686883] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039249 [35327.783147] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276385 [35330.687411] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031137 [35330.688179] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035346 [35391.803491] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [35692.054148] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [35692.168313] Lustre: Skipped 12 previous similar messages [35757.797211] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [35915.334530] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803c5583850 x1586704102710544/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:295/0 lens 608/0 e 0 to 0 dl 1513246040 ref 2 fl New:/2/ffffffff rc 0/-1 [35915.690352] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 782 previous similar messages [35920.388266] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [35920.506627] Lustre: Skipped 78 previous similar messages [35920.573232] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [35920.694465] Lustre: Skipped 79 previous similar messages [35920.761599] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337474 [35921.878556] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632097 [35925.692125] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055378 [35925.877917] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624321 [35926.878152] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612641 [35926.878422] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610721 [35928.677127] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039281 [35928.772570] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276417 [35931.678162] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031169 [35931.678780] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035378 [36342.042928] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [36342.157027] Lustre: Skipped 12 previous similar messages [36489.784644] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [36516.412221] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8800ab74bc50 x1586704106503136/t0(0) o4->13292baa-da7b-6c1c-3d65-2f4969d8fa28@192.168.1.134@o2ib:142/0 lens 608/0 e 0 to 0 dl 1513246642 ref 2 fl New:/2/ffffffff rc 0/-1 [36516.517668] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [36521.595174] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [36521.632366] Lustre: Skipped 79 previous similar messages [36521.652397] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [36521.690259] Lustre: Skipped 78 previous similar messages [36521.749488] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337506 [36522.860316] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632129 [36526.729639] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055410 [36526.858628] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624353 [36527.859437] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612673 [36527.859787] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610753 [36529.656732] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039313 [36529.762205] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276449 [36532.657510] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031201 [36532.658015] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035410 [36992.031613] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [36992.145789] Lustre: Skipped 12 previous similar messages [37116.529904] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803a9626050 x1586704102711920/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:742/0 lens 608/0 e 0 to 0 dl 1513247242 ref 2 fl New:/2/ffffffff rc 0/-1 [37116.635273] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [37122.583945] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [37122.621184] Lustre: Skipped 79 previous similar messages [37122.641267] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [37122.664462] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337538 [37122.713872] Lustre: Skipped 80 previous similar messages [37124.779676] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632161 [37127.679084] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055442 [37128.197363] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624385 [37129.197235] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612705 [37129.197535] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610785 [37130.646806] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039345 [37130.751880] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276481 [37133.648088] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035442 [37133.648449] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031233 [37159.773129] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [37525.766845] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [37642.020428] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [37642.053820] Lustre: Skipped 12 previous similar messages [37717.607589] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8802efa03850 x1586704102710928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:588/0 lens 608/0 e 0 to 0 dl 1513247843 ref 2 fl New:/2/ffffffff rc 0/-1 [37717.961918] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [37723.572647] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [37723.669130] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [37723.669133] Lustre: Skipped 77 previous similar messages [37723.669961] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337570 [37724.007784] Lustre: Skipped 79 previous similar messages [37725.701017] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632193 [37729.106823] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055474 [37729.140928] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624417 [37730.141944] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612737 [37730.142212] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610817 [37731.637582] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039377 [37731.741548] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276513 [37734.638553] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031265 [37734.639370] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035474 [38292.009108] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [38292.123224] Lustre: Skipped 12 previous similar messages [38318.685257] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803d4c78850 x1586704102713184/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:434/0 lens 608/0 e 0 to 0 dl 1513248444 ref 2 fl New:H/2/ffffffff rc 0/-1 [38319.042046] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 794 previous similar messages [38324.561302] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [38324.562720] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [38324.562723] Lustre: Skipped 78 previous similar messages [38324.563695] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337602 [38324.996488] Lustre: Skipped 77 previous similar messages [38325.869100] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632225 [38329.867541] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624449 [38329.990691] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055506 [38330.868459] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612769 [38330.868817] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610849 [38332.628163] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039409 [38332.731155] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276545 [38335.628937] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035506 [38335.629424] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031297 [38919.698946] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff880078a45c50 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:280/0 lens 608/0 e 0 to 0 dl 1513249045 ref 2 fl New:/2/ffffffff rc 0/-1 [38920.054565] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 776 previous similar messages [38925.550065] Lustre: soaked-OST0005: Client 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) reconnecting [38925.601164] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [38925.601167] Lustre: Skipped 79 previous similar messages [38925.602081] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337634 [38925.985536] Lustre: Skipped 79 previous similar messages [38926.742778] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [38926.871210] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632257 [38930.647455] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055538 [38930.816154] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624481 [38931.817041] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612801 [38931.817408] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610881 [38933.618764] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039441 [38933.720519] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276577 [38936.619373] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035538 [38936.620071] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031329 [38941.997930] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [38942.111991] Lustre: Skipped 12 previous similar messages [39292.736502] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [39292.772570] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [39520.776601] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803c785b050 x1586704102712848/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:126/0 lens 608/0 e 0 to 0 dl 1513249646 ref 2 fl New:/2/ffffffff rc 0/-1 [39520.881989] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [39526.517958] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [39526.538853] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [39526.538856] Lustre: Skipped 77 previous similar messages [39526.610777] Lustre: Skipped 78 previous similar messages [39526.631605] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337666 [39527.815401] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632289 [39531.814769] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624513 [39531.822710] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055570 [39532.814999] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612833 [39532.815227] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610913 [39534.609108] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039473 [39534.710214] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276609 [39537.609949] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031361 [39537.610647] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035570 [39591.986547] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [39592.019795] Lustre: Skipped 12 previous similar messages [40024.723923] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [40024.848301] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [40121.854277] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803ec0a8c50 x1586704102712848/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:727/0 lens 608/0 e 0 to 0 dl 1513250247 ref 2 fl New:/2/ffffffff rc 0/-1 [40121.896342] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [40127.445347] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [40127.458832] Lustre: Skipped 77 previous similar messages [40127.467273] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [40127.480805] Lustre: Skipped 78 previous similar messages [40127.489956] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337698 [40128.807300] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632321 [40132.806364] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624545 [40132.885141] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055602 [40133.807304] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612865 [40133.807591] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610945 [40135.599593] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039505 [40135.699912] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276641 [40138.600527] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031393 [40138.600986] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035602 [40241.975329] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [40242.008626] Lustre: Skipped 12 previous similar messages [40722.931971] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803ab6b6450 x1586704102711920/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:573/0 lens 608/0 e 0 to 0 dl 1513250848 ref 2 fl New:/2/ffffffff rc 0/-1 [40723.287293] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 794 previous similar messages [40728.433078] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [40728.516205] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [40728.516207] Lustre: Skipped 78 previous similar messages [40728.748909] Lustre: Skipped 81 previous similar messages [40728.815682] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337730 [40729.887478] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632353 [40733.808215] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055634 [40733.887007] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624577 [40734.887009] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612897 [40734.887305] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10610977 [40736.590184] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039537 [40736.689311] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276673 [40739.590983] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031425 [40739.591556] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035634 [40891.964147] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [40891.997376] Lustre: Skipped 12 previous similar messages [41060.706126] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [41324.009640] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88004ed03c50 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:419/0 lens 608/0 e 0 to 0 dl 1513251449 ref 2 fl New:/2/ffffffff rc 0/-1 [41324.114647] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 776 previous similar messages [41329.310744] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [41329.345241] Lustre: Skipped 76 previous similar messages [41329.365262] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [41329.400193] Lustre: Skipped 75 previous similar messages [41329.421168] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337762 [41330.825088] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632385 [41334.765606] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055666 [41334.824898] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624609 [41335.824604] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10611009 [41335.824764] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612929 [41337.580459] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039569 [41337.678918] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276705 [41340.581388] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031457 [41340.581908] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035666 [41426.699843] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [41541.952824] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [41541.965860] Lustre: Skipped 12 previous similar messages [41792.693556] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [41925.087330] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8802202e7450 x1586704102711920/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:265/0 lens 608/0 e 0 to 0 dl 1513252050 ref 2 fl New:/2/ffffffff rc 0/-1 [41925.129672] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages [41930.332355] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [41930.345878] Lustre: Skipped 79 previous similar messages [41930.354358] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [41930.367910] Lustre: Skipped 79 previous similar messages [41930.377093] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337794 [41931.757646] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632417 [41935.732723] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055698 [41935.757284] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624641 [41936.757396] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612961 [41936.757673] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10611041 [41938.571027] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039601 [41938.668761] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276737 [41941.571906] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031489 [41941.572550] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035698 [42191.941513] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [42191.974885] Lustre: Skipped 12 previous similar messages [42526.164992] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803c89d6450 x1586704102710544/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:111/0 lens 608/0 e 0 to 0 dl 1513252651 ref 2 fl New:/2/ffffffff rc 0/-1 [42526.207704] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 784 previous similar messages [42531.305948] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [42531.319562] Lustre: Skipped 78 previous similar messages [42531.328117] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [42531.341789] Lustre: Skipped 79 previous similar messages [42531.351092] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337826 [42532.860988] Lustre: soaked-OST0005: deleting orphan objects from 0x440000401:10630651 to 0x440000401:10632449 [42536.712645] Lustre: soaked-OST0005: deleting orphan objects from 0x440000400:9054231 to 0x440000400:9055730 [42536.860972] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000402:10622846 to 0x5c0000402:10624673 [42537.861510] Lustre: soaked-OST0011: deleting orphan objects from 0x740000402:10611070 to 0x740000402:10612993 [42537.861773] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000401:10609153 to 0x8c0000401:10611073 [42539.561130] Lustre: soaked-OST000b: deleting orphan objects from 0x5c0000401:9038135 to 0x5c0000401:9039633 [42539.657898] Lustre: soaked-OST0011: deleting orphan objects from 0x740000401:9275235 to 0x740000401:9276769 [42542.560904] Lustre: soaked-OST0011: deleting orphan objects from 0x740000400:9030022 to 0x740000400:9031521 [42542.561218] Lustre: soaked-OST0017: deleting orphan objects from 0x8c0000400:9034232 to 0x8c0000400:9035730 [42701.677879] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 0 seconds [42701.800800] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [42733.201346] LNet: Service thread pid 29726 was inactive for 200.34s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [42733.406519] Pid: 29726, comm: ll_ost00_015 [42733.456552] Call Trace: [42733.505479] [] schedule+0x29/0x70 [42733.565858] [] cv_wait_common+0x125/0x150 [spl] [42733.640767] [] ? autoremove_wake_function+0x0/0x40 [42733.718781] [] __cv_wait+0x15/0x20 [spl] [42733.786410] [] txg_wait_open+0xc3/0x110 [zfs] [42733.859207] [] dmu_tx_wait+0x3aa/0x3c0 [zfs] [42733.930920] [] dmu_tx_assign+0x91/0x490 [zfs] [42734.003647] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [42734.082618] [] ofd_destroy+0x248/0x940 [ofd] [42734.154308] [] ofd_destroy_by_fid+0x1e3/0x4a0 [ofd] [42734.233274] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [42734.314323] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [42734.397454] [] ofd_create_hdl+0x1abd/0x2090 [ofd] [42734.474351] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [42734.560657] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [42734.643848] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [42734.736417] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [42734.818528] [] ? default_wake_function+0x12/0x20 [42734.894409] [] ? __wake_up_common+0x58/0x90 [42734.965099] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [42735.040993] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [42735.116848] [] kthread+0xcf/0xe0 [42735.176075] [] ? kthread+0x0/0xe0 [42735.236272] [] ret_from_fork+0x58/0x90 [42735.301655] [] ? kthread+0x0/0xe0 [42735.361829] [42735.380377] LustreError: dumping log to /tmp/lustre-log.1513252856.29726 [42737.297287] LNet: Service thread pid 5754 was inactive for 200.43s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [42737.500591] Pid: 5754, comm: ll_ost00_001 [42737.549196] Call Trace: [42737.597411] [] schedule+0x29/0x70 [42737.657476] [] cv_wait_common+0x125/0x150 [spl] [42737.732087] [] ? autoremove_wake_function+0x0/0x40 [42737.809845] [] __cv_wait+0x15/0x20 [spl] [42737.877205] [] txg_wait_open+0xc3/0x110 [zfs] [42737.949756] [] dmu_tx_wait+0x3aa/0x3c0 [zfs] [42738.021273] [] dmu_tx_assign+0x91/0x490 [zfs] [42738.093860] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [42738.172653] [] ofd_destroy+0x248/0x940 [ofd] [42738.244155] [] ofd_destroy_by_fid+0x1e3/0x4a0 [ofd] [42738.322975] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [42738.403849] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [42738.486765] [] ofd_create_hdl+0x1abd/0x2090 [ofd] [42738.563455] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [42738.649515] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [42738.732458] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [42738.824742] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [42738.906626] [] ? default_wake_function+0x12/0x20 [42738.982264] [] ? __wake_up_common+0x58/0x90 [42739.052734] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [42739.128402] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [42739.204023] [] kthread+0xcf/0xe0 [42739.263001] [] ? kthread+0x0/0xe0 [42739.323017] [] ret_from_fork+0x58/0x90 [42739.388231] [] ? kthread+0x0/0xe0 [42739.448241] [42739.466638] LustreError: dumping log to /tmp/lustre-log.1513252860.5754 [42739.599825] LNet: Service thread pid 29728 was inactive for 201.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [42739.804051] Pid: 29728, comm: ll_ost00_017 [42739.853692] Call Trace: [42739.901900] [] schedule+0x29/0x70 [42739.961950] [] cv_wait_common+0x125/0x150 [spl] [42740.036561] [] ? autoremove_wake_function+0x0/0x40 [42740.114296] [] __cv_wait+0x15/0x20 [spl] [42740.181661] [] txg_wait_open+0xc3/0x110 [zfs] [42740.254186] [] ? getrawmonotonic64+0x2e/0xc0 [42740.325718] [] dmu_tx_wait+0x3aa/0x3c0 [zfs] [42740.397232] [] dmu_tx_assign+0x91/0x490 [zfs] [42740.469775] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [42740.548565] [] ofd_destroy+0x248/0x940 [ofd] [42740.620064] [] ofd_destroy_by_fid+0x1e3/0x4a0 [ofd] [42740.698854] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [42740.779700] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [42740.862599] [] ofd_create_hdl+0x1abd/0x2090 [ofd] [42740.939312] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [42741.025380] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [42741.108299] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [42741.200583] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [42741.282445] [] ? default_wake_function+0x12/0x20 [42741.358092] [] ? __wake_up_common+0x58/0x90 [42741.444550] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [42741.536191] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [42741.627793] [] kthread+0xcf/0xe0 [42741.691340] [] ? kthread+0x0/0xe0 [42741.751371] [] ret_from_fork+0x58/0x90 [42741.816602] [] ? kthread+0x0/0xe0 [42741.876624] [42741.895037] LNet: Service thread pid 27490 was inactive for 202.24s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [42742.099287] Pid: 27490, comm: ll_ost01_027 [42742.148948] Call Trace: [42742.197147] [] schedule+0x29/0x70 [42742.257195] [] cv_wait_common+0x125/0x150 [spl] [42742.331800] [] ? autoremove_wake_function+0x0/0x40 [42742.409537] [] __cv_wait+0x15/0x20 [spl] [42742.476892] [] txg_wait_open+0xc3/0x110 [zfs] [42742.549442] [] dmu_tx_wait+0x3aa/0x3c0 [zfs] [42742.620949] [] dmu_tx_assign+0x91/0x490 [zfs] [42742.693494] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [42742.772280] [] ofd_destroy+0x248/0x940 [ofd] [42742.843782] [] ofd_destroy_by_fid+0x1e3/0x4a0 [ofd] [42742.922577] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [42743.003446] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [42743.086351] [] ofd_create_hdl+0x1abd/0x2090 [ofd] [42743.163032] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [42743.249124] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [42743.332062] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [42743.424370] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [42743.506255] [] ? default_wake_function+0x12/0x20 [42743.581895] [] ? __wake_up_common+0x58/0x90 [42743.652355] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [42743.727993] [] ? __switch_to+0xd7/0x510 [42743.794292] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [42743.869933] [] kthread+0xcf/0xe0 [42743.928926] [] ? kthread+0x0/0xe0 [42743.988960] [] ret_from_fork+0x58/0x90 [42744.054193] [] ? kthread+0x0/0xe0 [42744.114219] [42744.132637] Pid: 5756, comm: ll_ost01_000 [42744.181218] Call Trace: [42744.229356] [] schedule+0x29/0x70 [42744.289363] [] cv_wait_common+0x125/0x150 [spl] [42744.363930] [] ? autoremove_wake_function+0x0/0x40 [42744.441619] [] __cv_wait+0x15/0x20 [spl] [42744.508923] [] txg_wait_open+0xc3/0x110 [zfs] [42744.581424] [] dmu_tx_wait+0x3aa/0x3c0 [zfs] [42744.652881] [] dmu_tx_assign+0x91/0x490 [zfs] [42744.725367] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [42744.804093] [] ofd_destroy+0x248/0x940 [ofd] [42744.875444] [] ofd_destroy_by_fid+0x1e3/0x4a0 [ofd] [42744.954195] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [42745.035031] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [42745.117927] [] ofd_create_hdl+0x1abd/0x2090 [ofd] [42745.194594] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [42745.280614] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [42745.363504] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [42745.455756] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [42745.537597] [] ? default_wake_function+0x12/0x20 [42745.613199] [] ? __wake_up_common+0x58/0x90 [42745.683619] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [42745.759242] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [42745.834837] [] kthread+0xcf/0xe0 [42745.893784] [] ? kthread+0x0/0xe0 [42745.953772] [] ret_from_fork+0x58/0x90 [42746.018969] [] ? kthread+0x0/0xe0 [42746.078954] [42746.097330] LNet: Service thread pid 148367 was inactive for 203.53s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [42746.253694] LNet: Skipped 8 previous similar messages [42764.676798] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [42841.929776] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [42841.940956] Lustre: Skipped 12 previous similar messages [43127.242583] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803f8602050 x1586704102710544/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:712/0 lens 608/0 e 0 to 0 dl 1513253252 ref 2 fl New:/2/ffffffff rc 0/-1 [43127.342051] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 785 previous similar messages [43132.254132] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [43132.286909] Lustre: Skipped 79 previous similar messages [43132.305250] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [43132.338753] Lustre: Skipped 79 previous similar messages [43132.357830] Lustre: soaked-OST0005: deleting orphan objects from 0x440000402:9336292 to 0x440000402:9337858 [43491.916570] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [43491.927904] Lustre: Skipped 12 previous similar messages [43496.664231] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 7 seconds [43727.360275] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8803f5ec1450 x1586704102712928/t0(0) o4->8c32ba31-522a-4ad5-f163-829b7c7fd32a@192.168.1.133@o2ib:558/0 lens 608/0 e 0 to 0 dl 1513253853 ref 2 fl New:/2/ffffffff rc 0/-1 [43727.396534] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 794 previous similar messages [43733.209747] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [43733.221596] Lustre: Skipped 79 previous similar messages [43733.228534] Lustre: soaked-OST0005: Connection restored to soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) [43733.240614] Lustre: Skipped 79 previous similar messages [43799.659021] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [44102.653825] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.115@o2ib: 1 seconds [44102.776681] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Skipped 1 previous similar message [44141.903111] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [44142.015836] Lustre: Skipped 12 previous similar messages [44328.437986] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff88026e303c50 x1586704105272848/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:404/0 lens 608/0 e 0 to 0 dl 1513254454 ref 2 fl New:/2/ffffffff rc 0/-1 [44328.787648] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 793 previous similar messages [44334.389298] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [44334.443246] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [44334.443247] Lustre: Skipped 77 previous similar messages [44334.701044] Lustre: Skipped 80 previous similar messages [44335.733813] LNet: Service thread pid 29742 was inactive for 1200.90s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [44335.940230] LNet: Skipped 1 previous similar message [44336.000856] Pid: 29742, comm: ll_ost00_023 [44336.051019] Call Trace: [44336.100297] [] schedule_preempt_disabled+0x29/0x70 [44336.178547] [] __mutex_lock_slowpath+0xc7/0x1d0 [44336.253676] [] mutex_lock+0x1f/0x2f [44336.316312] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [44336.392504] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [44336.479054] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [44336.565590] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [44336.646887] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [44336.730240] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [44336.822932] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [44336.905188] [] ? default_wake_function+0x12/0x20 [44336.981199] [] ? __wake_up_common+0x58/0x90 [44337.052012] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [44337.128018] [] ? __switch_to+0xd7/0x510 [44337.194655] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [44337.270618] [] kthread+0xcf/0xe0 [44337.329916] [] ? kthread+0x0/0xe0 [44337.390240] [] ret_from_fork+0x58/0x90 [44337.455738] [] ? kthread+0x0/0xe0 [44337.516039] [44337.534689] LustreError: dumping log to /tmp/lustre-log.1513254458.29742 [44337.745264] LNet: Service thread pid 9119 was inactive for 1205.40s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [44337.745265] Pid: 9119, comm: ll_ost01_014 [44337.745265] Call Trace: [44337.745267] [] schedule+0x29/0x70 [44337.745280] [] cv_wait_common+0x125/0x150 [spl] [44337.745281] [] ? autoremove_wake_function+0x0/0x40 [44337.745285] [] __cv_wait+0x15/0x20 [spl] [44337.745327] [] txg_wait_open+0xc3/0x110 [zfs] [44337.745344] [] dmu_tx_wait+0x3aa/0x3c0 [zfs] [44337.745359] [] dmu_tx_assign+0x91/0x490 [zfs] [44337.745365] [] osd_trans_start+0xaa/0x3c0 [osd_zfs] [44337.745373] [] ofd_destroy+0x248/0x940 [ofd] [44337.745377] [] ofd_destroy_by_fid+0x1e3/0x4a0 [ofd] [44337.745402] [] ? ldlm_blocking_ast+0x0/0x170 [ptlrpc] [44337.745423] [] ? ldlm_completion_ast+0x0/0x920 [ptlrpc] [44337.745427] [] ofd_create_hdl+0x1abd/0x2090 [ofd] [44337.745451] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [44337.745480] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [44337.745504] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [44337.745528] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [44337.745554] [] ? keys_fini+0xb1/0x1d0 [obdclass] [44337.745578] [] ? ptlrpc_server_drop_request+0x431/0x6c0 [ptlrpc] [44337.745602] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [44337.745625] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [44337.745626] [] kthread+0xcf/0xe0 [44337.745627] [] ? kthread+0x0/0xe0 [44337.745629] [] ret_from_fork+0x58/0x90 [44337.745630] [] ? kthread+0x0/0xe0 [44337.745630] [44339.829743] LNet: Service thread pid 103074 was inactive for 1201.15s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [44339.829744] Pid: 103074, comm: ll_ost01_048 [44339.829744] Call Trace: [44339.829748] [] schedule_preempt_disabled+0x29/0x70 [44339.829750] [] __mutex_lock_slowpath+0xc7/0x1d0 [44339.829752] [] mutex_lock+0x1f/0x2f [44339.829759] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [44339.829793] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [44339.829818] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [44339.829842] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [44339.829872] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [44339.829897] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [44339.829921] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [44339.829923] [] ? default_wake_function+0x12/0x20 [44339.829924] [] ? __wake_up_common+0x58/0x90 [44339.829949] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [44339.829973] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [44339.829974] [] kthread+0xcf/0xe0 [44339.829975] [] ? kthread+0x0/0xe0 [44339.829977] [] ret_from_fork+0x58/0x90 [44339.829978] [] ? kthread+0x0/0xe0 [44339.829978] [44339.829979] LustreError: dumping log to /tmp/lustre-log.1513254460.103074 [44339.857558] Pid: 54982, comm: ll_ost00_039 [44339.857559] Call Trace: [44339.857561] [] schedule_preempt_disabled+0x29/0x70 [44339.857563] [] __mutex_lock_slowpath+0xc7/0x1d0 [44339.857565] [] mutex_lock+0x1f/0x2f [44339.857569] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [44339.857596] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [44339.857622] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [44339.857645] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [44339.857675] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [44339.857703] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [44339.857733] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [44339.857735] [] ? default_wake_function+0x12/0x20 [44339.857736] [] ? __wake_up_common+0x58/0x90 [44339.857763] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [44339.857787] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [44339.857788] [] kthread+0xcf/0xe0 [44339.857789] [] ? kthread+0x0/0xe0 [44339.857790] [] ret_from_fork+0x58/0x90 [44339.857791] [] ? kthread+0x0/0xe0 [44339.857792] [44339.857794] Pid: 29756, comm: ll_ost00_034 [44339.857794] Call Trace: [44339.857796] [] schedule_preempt_disabled+0x29/0x70 [44339.857798] [] __mutex_lock_slowpath+0xc7/0x1d0 [44339.857800] [] mutex_lock+0x1f/0x2f [44339.857805] [] ofd_create_hdl+0xdcb/0x2090 [ofd] [44339.857830] [] ? lustre_msg_add_version+0x27/0xa0 [ptlrpc] [44339.857854] [] ? lustre_pack_reply_v2+0x14f/0x280 [ptlrpc] [44339.857879] [] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [44339.857908] [] tgt_request_handle+0x925/0x1370 [ptlrpc] [44339.857933] [] ptlrpc_server_handle_request+0x236/0xa90 [ptlrpc] [44339.857957] [] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] [44339.857958] [] ? default_wake_function+0x12/0x20 [44339.857960] [] ? __wake_up_common+0x58/0x90 [44339.857985] [] ptlrpc_main+0xa92/0x1e40 [ptlrpc] [44339.858011] [] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] [44339.858012] [] kthread+0xcf/0xe0 [44339.858013] [] ? kthread+0x0/0xe0 [44339.858014] [] ret_from_fork+0x58/0x90 [44339.858015] [] ? kthread+0x0/0xe0 [44339.858016] [44343.925683] LNet: Service thread pid 103097 was inactive for 1202.37s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [44343.925685] LNet: Skipped 3 previous similar messages [44343.925687] LustreError: dumping log to /tmp/lustre-log.1513254464.103097 [44348.021604] LNet: Service thread pid 72214 was inactive for 1203.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. [44348.178864] LNet: Skipped 2 previous similar messages [44348.240314] LustreError: dumping log to /tmp/lustre-log.1513254468.72214 [44771.642327] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.114@o2ib: 0 seconds [44791.889635] Lustre: soaked-OST0005: Export ffff88074f3a3000 already connecting from 192.168.1.108@o2ib [44792.001974] Lustre: Skipped 12 previous similar messages [44929.515633] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply req@ffff8804149d5450 x1586704105272848/t0(0) o4->7df9a58e-df80-cd01-0d09-5737d3c03bbe@192.168.1.131@o2ib:250/0 lens 608/0 e 0 to 0 dl 1513255055 ref 2 fl New:/2/ffffffff rc 0/-1 [44929.613566] Lustre: 136835:0:(service.c:1346:ptlrpc_at_send_early_reply()) Skipped 795 previous similar messages [44935.379406] Lustre: soaked-OST0005: Client soaked-MDT0003-mdtlov_UUID (at 192.168.1.111@o2ib) reconnecting [44935.412601] Lustre: Skipped 77 previous similar messages [44935.429973] Lustre: soaked-OST0005: Connection restored to 6943d9de-cea7-4a53-1661-77d616b076c6 (at 192.168.1.136@o2ib) [44935.429975] Lustre: Skipped 79 previous similar messages [44940.659687] gmain invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44940.685141] gmain cpuset=/ mems_allowed=0-1 [44940.699853] CPU: 10 PID: 2945 Comm: gmain Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44940.735748] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44940.773670] ffff8800b12d4f10 00000000b91ef903 ffff880411287988 ffffffff816a3e2d [44940.799046] ffff880411287a18 ffffffff8169f246 ffffffff810e93ac ffff880410fb97c0 [44940.824448] ffff880410fb97d8 0000000000000202 0000000000000202 ffff880411287a08 [44940.849853] Call Trace: [44940.858910] [] dump_stack+0x19/0x1b [44940.876837] [] dump_header+0x90/0x229 [44940.895326] [] ? ktime_get_ts64+0x4c/0xf0 [44940.914962] [] ? delayacct_end+0x8f/0xb0 [44940.934307] [] oom_kill_process+0x254/0x3d0 [44940.954512] [] ? oom_unkillable_task+0xcd/0x120 [44940.975861] [] ? find_lock_task_mm+0x56/0xc0 [44940.996357] [] out_of_memory+0x4b6/0x4f0 [44941.015701] [] __alloc_pages_slowpath+0x5d6/0x724 [44941.037591] [] __alloc_pages_nodemask+0x405/0x420 [44941.059496] [] alloc_pages_vma+0xb5/0x200 [44941.079116] [] read_swap_cache_async+0xed/0x160 [44941.100448] [] swapin_readahead+0xa8/0x110 [44941.120355] [] handle_mm_fault+0xadb/0xfa0 [44941.140262] [] __do_page_fault+0x154/0x450 [44941.160165] [] do_page_fault+0x35/0x90 [44941.178931] [] page_fault+0x28/0x30 [44941.196833] Mem-Info: [44941.205292] active_anon:35 inactive_anon:0 isolated_anon:0 active_file:0 inactive_file:80 isolated_file:0 unevictable:6965 dirty:0 writeback:11 unstable:0 slab_reclaimable:9803 slab_unreclaimable:234987 mapped:1718 shmem:0 pagetables:1777 bounce:0 free:39369 free_pcp:20 free_cma:0 [44941.317280] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44941.457405] lowmem_reserve[]: 0 2580 15620 15620 [44941.473633] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:4kB inactive_anon:4kB active_file:8kB inactive_file:4kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2240kB slab_unreclaimable:59756kB kernel_stack:1056kB pagetables:360kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:118 all_unreclaimable? yes [44941.623315] lowmem_reserve[]: 0 0 13040 13040 [44941.638694] Node 0 Normal free:36872kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:32kB active_file:0kB inactive_file:372kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:4120kB shmem:0kB slab_reclaimable:16428kB slab_unreclaimable:312080kB kernel_stack:10832kB pagetables:3212kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:973 all_unreclaimable? yes [44941.793202] lowmem_reserve[]: 0 0 0 0 [44941.806201] Node 1 Normal free:45552kB min:45728kB low:57160kB high:68592kB active_anon:140kB inactive_anon:0kB active_file:44kB inactive_file:184kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:44kB mapped:2180kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:568112kB kernel_stack:19024kB pagetables:3536kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1148 all_unreclaimable? yes [44941.962556] lowmem_reserve[]: 0 0 0 0 [44941.975591] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44942.020723] Node 0 DMA32: 400*4kB (UEM) 615*8kB (UEM) 478*16kB (UEM) 418*32kB (UEM) 172*64kB (UEM) 22*128kB (EM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 4*2048kB (M) 0*4096kB = 59544kB [44942.077164] Node 0 Normal: 183*4kB (UEM) 883*8kB (UEM) 494*16kB (UEM) 417*32kB (UEM) 118*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37876kB [44942.128198] Node 1 Normal: 1054*4kB (UEM) 729*8kB (UEM) 488*16kB (UEM) 382*32kB (UEM) 265*64kB (UEM) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 48704kB [44942.179875] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44942.209631] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44942.238520] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44942.268263] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44942.297135] 2053 total pagecache pages [44942.310568] 0 pages in swap cache [44942.322563] Swap cache stats: add 69725, delete 69725, find 14748/19926 [44942.345438] Free swap = 16207300kB [44942.358023] Total swap = 16319484kB [44942.370603] 8369063 pages RAM [44942.381462] 0 pages HighMem/MovableOnly [44942.395193] 241261 pages reserved [44942.407185] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44942.434089] [ 1003] 0 1003 24572 213 51 83 0 systemd-journal [44942.464139] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44942.491892] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44942.521363] [ 1081] 0 1081 1588718 4010 133 0 -1000 multipathd [44942.549986] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44942.577434] [ 2588] 81 2588 9199 179 21 137 -900 dbus-daemon [44942.606290] [ 2619] 0 2619 67353 264 73 4999 0 rsyslogd [44942.634268] [ 2621] 0 2621 64917 325 78 276 0 sssd [44942.661086] [ 2622] 999 2622 136594 208 61 1634 0 polkitd [44942.688739] [ 2624] 0 2624 118083 236 83 763 0 NetworkManager [44942.718383] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44942.745999] [ 2637] 0 2637 5461 187 16 146 0 irqbalance [44942.774468] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44942.801751] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44942.829583] [ 2938] 0 2938 93409 307 124 637 0 sssd_be [44942.857109] [ 2940] 0 2940 66463 289 84 378 0 sssd_nss [44942.884900] [ 2941] 0 2941 61252 284 74 233 0 sssd_pam [44942.912666] [ 2942] 0 2942 59613 270 71 220 0 sssd_ssh [44942.940407] [ 2943] 0 2943 68163 270 86 332 0 sssd_pac [44942.968120] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44942.995832] [ 2955] 0 2955 6051 224 16 80 0 systemd-logind [44943.025242] [ 3687] 0 3687 140597 288 93 2769 0 tuned [44943.052053] [ 3689] 996 3689 24211 171 21 130 0 munged [44943.079131] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44943.106185] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44943.132644] [ 3859] 0 3859 22907 170 44 263 0 master [44943.159671] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44943.186104] [ 3883] 0 3883 6464 163 19 52 0 atd [44943.212230] [ 3900] 0 3900 151290 201 68 378 0 automount [44943.240052] [ 3922] 0 3922 31559 188 19 157 0 crond [44943.266710] [ 3971] 0 3971 27511 156 12 32 0 agetty [44943.293628] [ 3975] 0 3975 27511 160 10 33 0 agetty [44943.320561] [147960] 89 147960 25474 209 46 256 0 pickup [44943.348010] [151569] 0 151569 40498 272 78 312 0 sshd [44943.374861] [151570] 74 151570 27583 151 56 273 0 sshd [44943.401692] Out of memory: Kill process 2619 (rsyslogd) score 0 or sacrifice child [44943.427112] Killed process 2619 (rsyslogd) total-vm:269412kB, anon-rss:0kB, file-rss:1056kB, shmem-rss:0kB [44943.533981] gmain invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [44943.559257] gmain cpuset=/ mems_allowed=0-1 [44943.573890] CPU: 12 PID: 2945 Comm: gmain Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44943.609728] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44943.647603] ffff8800b12d4f10 00000000b91ef903 ffff880411287920 ffffffff816a3e2d [44943.672967] ffff8804112879b0 ffffffff8169f246 ffffffff810e93ac ffff880410fb97c0 [44943.698336] ffff880410fb97d8 0000000000000202 0000000000000202 ffff8804112879a0 [44943.723706] Call Trace: [44943.732705] [] dump_stack+0x19/0x1b [44943.750588] [] dump_header+0x90/0x229 [44943.769023] [] ? ktime_get_ts64+0x4c/0xf0 [44943.788591] [] ? delayacct_end+0x8f/0xb0 [44943.807878] [] oom_kill_process+0x254/0x3d0 [44943.828006] [] ? oom_unkillable_task+0xcd/0x120 [44943.849251] [] ? find_lock_task_mm+0x56/0xc0 [44943.869639] [] out_of_memory+0x4b6/0x4f0 [44943.888879] [] __alloc_pages_slowpath+0x5d6/0x724 [44943.910699] [] __alloc_pages_nodemask+0x405/0x420 [44943.932518] [] alloc_pages_current+0x98/0x110 [44943.953188] [] __page_cache_alloc+0x97/0xb0 [44943.973284] [] filemap_fault+0x170/0x410 [44943.992539] [] ext4_filemap_fault+0x36/0x50 [ext4] [44944.014632] [] __do_fault+0x52/0xe0 [44944.032443] [] ? do_wp_page+0x185/0x5c0 [44944.051400] [] do_read_fault.isra.44+0x4b/0x130 [44944.072336] [] handle_mm_fault+0x691/0xfa0 [44944.091844] [] __do_page_fault+0x154/0x450 [44944.111350] [] do_page_fault+0x35/0x90 [44944.129713] [] page_fault+0x28/0x30 [44944.147218] Mem-Info: [44944.155283] active_anon:0 inactive_anon:2 isolated_anon:0 active_file:177 inactive_file:73 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9802 slab_unreclaimable:234986 mapped:1558 shmem:0 pagetables:1777 bounce:0 free:39303 free_pcp:0 free_cma:0 [44944.264894] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44944.403899] lowmem_reserve[]: 0 2580 15620 15620 [44944.419879] Node 0 DMA32 free:59392kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2240kB slab_unreclaimable:59756kB kernel_stack:1056kB pagetables:360kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:390 all_unreclaimable? yes [44944.569661] lowmem_reserve[]: 0 0 13040 13040 [44944.584886] Node 0 Normal free:36900kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:128kB inactive_file:12kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16424kB slab_unreclaimable:312112kB kernel_stack:10832kB pagetables:3212kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:280 all_unreclaimable? yes [44944.739175] lowmem_reserve[]: 0 0 0 0 [44944.752122] Node 1 Normal free:45384kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:204kB inactive_file:620kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1840kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:568076kB kernel_stack:19024kB pagetables:3536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1240 all_unreclaimable? yes [44944.907777] lowmem_reserve[]: 0 0 0 0 [44944.920815] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44944.965972] Node 0 DMA32: 388*4kB (UEM) 615*8kB (UEM) 478*16kB (UEM) 418*32kB (UEM) 172*64kB (UEM) 22*128kB (EM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 4*2048kB (M) 0*4096kB = 59496kB [44945.022412] Node 0 Normal: 187*4kB (EM) 759*8kB (UEM) 494*16kB (UEM) 417*32kB (UEM) 118*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36900kB [44945.073164] Node 1 Normal: 1052*4kB (EM) 728*8kB (UEM) 367*16kB (EM) 344*32kB (UEM) 266*64kB (UEM) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45600kB [44945.124284] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44945.154047] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44945.182945] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44945.212692] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44945.241570] 1802 total pagecache pages [44945.255002] 0 pages in swap cache [44945.267000] Swap cache stats: add 69769, delete 69769, find 14759/19943 [44945.289881] Free swap = 16207300kB [44945.302440] Total swap = 16319484kB [44945.315022] 8369063 pages RAM [44945.325883] 0 pages HighMem/MovableOnly [44945.339605] 241261 pages reserved [44945.351601] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44945.378503] [ 1003] 0 1003 24572 214 51 81 0 systemd-journal [44945.408555] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44945.436310] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44945.465783] [ 1081] 0 1081 1588718 4010 133 0 -1000 multipathd [44945.494403] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44945.521851] [ 2588] 81 2588 9199 179 21 137 -900 dbus-daemon [44945.550719] [ 2626] 0 2619 67353 285 73 4984 0 in:imjournal [44945.579855] [ 2621] 0 2621 64917 325 78 276 0 sssd [44945.606669] [ 2622] 999 2622 136594 208 61 1634 0 polkitd [44945.634323] [ 2624] 0 2624 118083 232 83 765 0 NetworkManager [44945.663968] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44945.691594] [ 2637] 0 2637 5461 187 16 146 0 irqbalance [44945.720048] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44945.747335] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44945.775171] [ 2938] 0 2938 93409 306 124 638 0 sssd_be [44945.802700] [ 2940] 0 2940 66463 289 84 378 0 sssd_nss [44945.830492] [ 2941] 0 2941 61252 284 74 233 0 sssd_pam [44945.858256] [ 2942] 0 2942 59613 270 71 220 0 sssd_ssh [44945.886013] [ 2943] 0 2943 68163 270 86 332 0 sssd_pac [44945.913728] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44945.941439] [ 2955] 0 2955 6051 224 16 80 0 systemd-logind [44945.970851] [ 3687] 0 3687 140597 288 93 2769 0 tuned [44945.997666] [ 3689] 996 3689 24211 171 21 130 0 munged [44946.024743] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44946.051800] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44946.078261] [ 3859] 0 3859 22907 170 44 263 0 master [44946.105288] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44946.131725] [ 3883] 0 3883 6464 163 19 52 0 atd [44946.157853] [ 3900] 0 3900 151290 201 68 378 0 automount [44946.185678] [ 3922] 0 3922 31559 188 19 157 0 crond [44946.212334] [ 3971] 0 3971 27511 156 12 32 0 agetty [44946.239254] [ 3975] 0 3975 27511 160 10 33 0 agetty [44946.266186] [147960] 89 147960 25474 209 46 256 0 pickup [44946.293636] [151569] 0 151569 40498 272 78 312 0 sshd [44946.320487] [151570] 74 151570 27583 151 56 273 0 sshd [44946.347320] Out of memory: Kill process 2626 (in:imjournal) score 0 or sacrifice child [44946.373883] Killed process 2626 (in:imjournal) total-vm:269412kB, anon-rss:0kB, file-rss:1140kB, shmem-rss:0kB [44946.543824] sssd_be invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44946.569455] sssd_be cpuset=/ mems_allowed=0-1 [44946.584476] CPU: 21 PID: 2938 Comm: sssd_be Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44946.620711] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44946.658385] ffff8800b12d3f40 00000000f3121f87 ffff88040e2a7988 ffffffff816a3e2d [44946.683519] ffff88040e2a7a18 ffffffff8169f246 ffffffff810e93ac ffff880410fb9520 [44946.708680] ffff880410fb9538 0000000000000202 0000000000000202 ffff88040e2a7a08 [44946.733756] Call Trace: [44946.742464] [] dump_stack+0x19/0x1b [44946.760027] [] dump_header+0x90/0x229 [44946.778227] [] ? ktime_get_ts64+0x4c/0xf0 [44946.797506] [] ? delayacct_end+0x8f/0xb0 [44946.816486] [] oom_kill_process+0x254/0x3d0 [44946.836373] [] ? oom_unkillable_task+0xcd/0x120 [44946.857396] [] ? find_lock_task_mm+0x56/0xc0 [44946.877615] [] out_of_memory+0x4b6/0x4f0 [44946.896687] [] __alloc_pages_slowpath+0x5d6/0x724 [44946.918326] [] __alloc_pages_nodemask+0x405/0x420 [44946.940061] [] alloc_pages_vma+0xb5/0x200 [44946.959361] [] ? end_swap_bio_write+0x80/0x80 [44946.979838] [] read_swap_cache_async+0xed/0x160 [44947.000878] [] ? task_numa_fault+0x8d0/0xbb0 [44947.020961] [] swapin_readahead+0xa8/0x110 [44947.040553] [] handle_mm_fault+0xadb/0xfa0 [44947.060176] [] __do_page_fault+0x154/0x450 [44947.079791] [] do_page_fault+0x35/0x90 [44947.098214] [] page_fault+0x28/0x30 [44947.115742] Mem-Info: [44947.123906] active_anon:0 inactive_anon:6 isolated_anon:0 active_file:258 inactive_file:0 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9802 slab_unreclaimable:234986 mapped:1547 shmem:0 pagetables:1777 bounce:0 free:39277 free_pcp:9 free_cma:0 [44947.233696] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44947.373046] lowmem_reserve[]: 0 2580 15620 15620 [44947.389080] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:12kB active_file:0kB inactive_file:16kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:584kB shmem:0kB slab_reclaimable:2240kB slab_unreclaimable:59756kB kernel_stack:1072kB pagetables:356kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:306 all_unreclaimable? no [44947.538822] lowmem_reserve[]: 0 0 13040 13040 [44947.554167] Node 0 Normal free:36832kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:128kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16420kB slab_unreclaimable:312112kB kernel_stack:10832kB pagetables:3184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:508 all_unreclaimable? yes [44947.709175] lowmem_reserve[]: 0 0 0 0 [44947.722299] Node 1 Normal free:45632kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:432kB inactive_file:360kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:568028kB kernel_stack:18992kB pagetables:3276kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6088 all_unreclaimable? no [44947.878257] lowmem_reserve[]: 0 0 0 0 [44947.891461] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44947.936885] Node 0 DMA32: 482*4kB (UEM) 619*8kB (UEM) 487*16kB (UEM) 420*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 3*1024kB (M) 4*2048kB (M) 0*4096kB = 59536kB [44947.993660] Node 0 Normal: 184*4kB (UEM) 514*8kB (UEM) 595*16kB (UEM) 424*32kB (UEM) 119*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36832kB [44948.044886] Node 1 Normal: 1051*4kB (UEM) 727*8kB (UEM) 369*16kB (UEM) 339*32kB (UEM) 269*64kB (UEM) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45652kB [44948.096904] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44948.126914] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44948.155979] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44948.185959] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44948.214984] 1795 total pagecache pages [44948.228490] 1 pages in swap cache [44948.240569] Swap cache stats: add 69915, delete 69914, find 14789/19999 [44948.263613] Free swap = 16227140kB [44948.276360] Total swap = 16319484kB [44948.289117] 8369063 pages RAM [44948.300144] 0 pages HighMem/MovableOnly [44948.314064] 241261 pages reserved [44948.326172] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44948.353130] [ 1003] 0 1003 24572 212 51 84 0 systemd-journal [44948.383272] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44948.411056] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44948.440612] [ 1081] 0 1081 1588718 4010 133 0 -1000 multipathd [44948.469370] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44948.496963] [ 2588] 81 2588 9199 179 21 137 -900 dbus-daemon [44948.525964] [ 2621] 0 2621 64917 325 78 276 0 sssd [44948.552941] [ 2622] 999 2622 136594 208 61 1634 0 polkitd [44948.580720] [ 2624] 0 2624 118083 229 83 770 0 NetworkManager [44948.610508] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44948.638255] [ 2637] 0 2637 5461 187 16 146 0 irqbalance [44948.666767] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44948.694186] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44948.722093] [ 2938] 0 2938 93409 306 124 638 0 sssd_be [44948.749725] [ 2940] 0 2940 66463 287 84 381 0 sssd_nss [44948.777635] [ 2941] 0 2941 61252 283 74 237 0 sssd_pam [44948.805536] [ 2942] 0 2942 59613 269 71 223 0 sssd_ssh [44948.833421] [ 2943] 0 2943 68163 269 86 336 0 sssd_pac [44948.861253] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44948.889056] [ 2955] 0 2955 6051 224 16 80 0 systemd-logind [44948.918538] [ 3687] 0 3687 140597 287 93 2772 0 tuned [44948.945457] [ 3689] 996 3689 24211 171 21 130 0 munged [44948.972639] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44948.999797] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44949.026425] [ 3859] 0 3859 22907 170 44 263 0 master [44949.053551] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44949.080126] [ 3883] 0 3883 6464 163 19 52 0 atd [44949.106335] [ 3900] 0 3900 151290 201 68 378 0 automount [44949.134292] [ 3922] 0 3922 31559 188 19 157 0 crond [44949.161077] [ 3971] 0 3971 27511 156 12 32 0 agetty [44949.188025] [ 3975] 0 3975 27511 160 10 33 0 agetty [44949.215078] [147960] 89 147960 25474 209 46 256 0 pickup [44949.242655] [151569] 0 151569 40498 272 78 312 0 sshd [44949.269623] [151570] 74 151570 27583 151 56 273 0 sshd [44949.296527] [151571] 0 151571 1 0 1 1 0 systemd-cgroups [44949.326600] Out of memory: Kill process 3687 (tuned) score 0 or sacrifice child [44949.351272] Killed process 3687 (tuned) total-vm:562388kB, anon-rss:0kB, file-rss:1148kB, shmem-rss:0kB [44949.563386] sssd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44949.588601] sssd cpuset=/ mems_allowed=0-1 [44949.602742] CPU: 12 PID: 2621 Comm: sssd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44949.637998] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44949.675528] ffff880429690fd0 0000000023efdee5 ffff8800b113f4a8 ffffffff816a3e2d [44949.700516] ffff8800b113f538 ffffffff8169f246 ffffffff810e93ac ffff88042b790f30 [44949.725509] ffff88042b790f48 0000000000000206 0000000000000206 ffff8800b113f528 [44949.750512] Call Trace: [44949.759159] [] dump_stack+0x19/0x1b [44949.776680] [] dump_header+0x90/0x229 [44949.794760] [] ? ktime_get_ts64+0x4c/0xf0 [44949.813974] [] ? delayacct_end+0x8f/0xb0 [44949.832880] [] oom_kill_process+0x254/0x3d0 [44949.852669] [] ? oom_unkillable_task+0xcd/0x120 [44949.873603] [] ? find_lock_task_mm+0x56/0xc0 [44949.893673] [] out_of_memory+0x4b6/0x4f0 [44949.912599] [] __alloc_pages_slowpath+0x5d6/0x724 [44949.934100] [] __alloc_pages_nodemask+0x405/0x420 [44949.955602] [] alloc_pages_vma+0xb5/0x200 [44949.974788] [] read_swap_cache_async+0xed/0x160 [44949.995716] [] ? zone_statistics+0x88/0xa0 [44950.015210] [] swapin_readahead+0xa8/0x110 [44950.034711] [] handle_mm_fault+0xadb/0xfa0 [44950.054209] [] ? dequeue_task_fair+0x41e/0x660 [44950.074853] [] ? sched_clock_cpu+0x85/0xc0 [44950.094559] [] __do_page_fault+0x154/0x450 [44950.114198] [] do_page_fault+0x35/0x90 [44950.132836] [] ? schedule_hrtimeout_range_clock+0xbe/0x150 [44950.157091] [] page_fault+0x28/0x30 [44950.174836] [] ? do_sys_poll+0x184/0x580 [44950.193924] [] ? do_sys_poll+0x145/0x580 [44950.212994] [] ? __schedule+0x39d/0x8b0 [44950.231785] [] ? bit_wait+0x50/0x50 [44950.249469] [] ? schedule+0x29/0x70 [44950.267200] [] ? schedule_timeout+0x239/0x2c0 [44950.287697] [] ? poll_select_copy_remaining+0x150/0x150 [44950.311073] [] ? poll_select_copy_remaining+0x150/0x150 [44950.334530] [] ? poll_select_copy_remaining+0x150/0x150 [44950.358002] [] ? poll_select_copy_remaining+0x150/0x150 [44950.381463] [] ? poll_select_copy_remaining+0x150/0x150 [44950.404895] [] ? poll_select_copy_remaining+0x150/0x150 [44950.428303] [] ? poll_select_copy_remaining+0x150/0x150 [44950.451703] [] ? poll_select_copy_remaining+0x150/0x150 [44950.475057] [] ? poll_select_copy_remaining+0x150/0x150 [44950.498415] [] SyS_poll+0x74/0x110 [44950.515804] [] system_call_fastpath+0x16/0x1b [44950.536281] Mem-Info: [44950.544444] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:42 inactive_file:203 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9801 slab_unreclaimable:234974 mapped:1547 shmem:0 pagetables:1705 bounce:0 free:39378 free_pcp:32 free_cma:0 [44950.654699] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44950.793566] lowmem_reserve[]: 0 2580 15620 15620 [44950.809501] Node 0 DMA32 free:59388kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2240kB slab_unreclaimable:59756kB kernel_stack:1072kB pagetables:360kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1464 all_unreclaimable? yes [44950.959349] lowmem_reserve[]: 0 0 13040 13040 [44950.974605] Node 0 Normal free:36832kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:184kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16420kB slab_unreclaimable:312112kB kernel_stack:10832kB pagetables:3184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:380 all_unreclaimable? yes [44951.129095] lowmem_reserve[]: 0 0 0 0 [44951.142297] Node 1 Normal free:45716kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:192kB inactive_file:620kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:568028kB kernel_stack:18960kB pagetables:3276kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1240 all_unreclaimable? yes [44951.298890] lowmem_reserve[]: 0 0 0 0 [44951.312155] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44951.357796] Node 0 DMA32: 445*4kB (UEM) 619*8kB (UEM) 487*16kB (UEM) 420*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 3*1024kB (M) 4*2048kB (M) 0*4096kB = 59388kB [44951.414719] Node 0 Normal: 184*4kB (UEM) 514*8kB (UEM) 596*16kB (UEM) 424*32kB (UEM) 119*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36848kB [44951.466434] Node 1 Normal: 1053*4kB (UEM) 727*8kB (UEM) 371*16kB (UEM) 339*32kB (UEM) 269*64kB (UEM) 13*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45692kB [44951.518797] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44951.548808] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44951.577948] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44951.607970] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44951.637169] 1785 total pagecache pages [44951.650998] 1 pages in swap cache [44951.663294] Swap cache stats: add 69981, delete 69980, find 14796/20017 [44951.686540] Free swap = 16227140kB [44951.699464] Total swap = 16319484kB [44951.712414] 8369063 pages RAM [44951.723619] 0 pages HighMem/MovableOnly [44951.737792] 241261 pages reserved [44951.750179] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44951.777436] [ 1003] 0 1003 24572 212 51 84 0 systemd-journal [44951.807818] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44951.835915] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44951.865717] [ 1081] 0 1081 1588718 4008 133 0 -1000 multipathd [44951.894788] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44951.922641] [ 2588] 81 2588 9199 179 21 137 -900 dbus-daemon [44951.951969] [ 2621] 0 2621 64917 325 78 276 0 sssd [44951.979288] [ 2622] 999 2622 136594 208 61 1634 0 polkitd [44952.007379] [ 2624] 0 2624 118083 229 83 771 0 NetworkManager [44952.037494] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44952.065570] [ 2637] 0 2637 5461 187 16 147 0 irqbalance [44952.094473] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44952.122167] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44952.150426] [ 2938] 0 2938 93409 329 124 621 0 sssd_be [44952.178368] [ 2940] 0 2940 66463 294 84 357 0 sssd_nss [44952.206568] [ 2941] 0 2941 61252 283 74 240 0 sssd_pam [44952.234721] [ 2942] 0 2942 59613 269 71 225 0 sssd_ssh [44952.262860] [ 2943] 0 2943 68163 269 86 337 0 sssd_pac [44952.290967] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44952.319026] [ 2955] 0 2955 6051 224 16 80 0 systemd-logind [44952.348827] [ 3689] 996 3689 24211 171 21 130 0 munged [44952.376300] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44952.403726] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44952.430540] [ 3859] 0 3859 22907 170 44 263 0 master [44952.457923] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44952.484683] [ 3883] 0 3883 6464 163 19 52 0 atd [44952.511138] [ 3900] 0 3900 151290 201 68 378 0 automount [44952.539302] [ 3922] 0 3922 31559 188 19 157 0 crond [44952.566129] [ 3971] 0 3971 27511 156 12 32 0 agetty [44952.593370] [ 3975] 0 3975 27511 160 10 33 0 agetty [44952.620566] [147960] 89 147960 25474 209 46 256 0 pickup [44952.648185] [151569] 0 151569 40498 272 78 312 0 sshd [44952.675313] [151570] 74 151570 27583 151 56 276 0 sshd [44952.702419] [151571] 0 151571 76 0 2 2 0 systemd-cgroups [44952.732568] [151572] 0 151572 41 0 2 1 0 systemd-cgroups [44952.762705] Out of memory: Kill process 3740 (gmain) score 0 or sacrifice child [44952.788848] ll_ost_out01_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [44952.816627] ll_ost_out01_00 cpuset=/ mems_allowed=1 [44952.833531] CPU: 30 PID: 32074 Comm: ll_ost_out01_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44952.873384] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44952.910906] ffff88080801bf40 000000004282fc3a ffff8803c00ab998 ffffffff816a3e2d [44952.935891] ffff8803c00aba28 ffffffff8169f246 ffffffff810e93ac ffff8804183a1600 [44952.960885] ffff8804183a1618 0000000000000206 0000000000000206 ffff8803c00aba18 [44952.985879] Call Trace: [44952.994519] [] dump_stack+0x19/0x1b [44953.012037] [] dump_header+0x90/0x229 [44953.030484] [] ? ktime_get_ts64+0x4c/0xf0 [44953.050067] [] ? delayacct_end+0x8f/0xb0 [44953.069362] [] oom_kill_process+0x254/0x3d0 [44953.089521] [] ? oom_unkillable_task+0xcd/0x120 [44953.110809] [] ? find_lock_task_mm+0x56/0xc0 [44953.131248] [] out_of_memory+0x4b6/0x4f0 [44953.150535] [] __alloc_pages_slowpath+0x5d6/0x724 [44953.172398] [] __alloc_pages_nodemask+0x405/0x420 [44953.194259] [] __vmalloc_node_range+0x12f/0x270 [44953.215619] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44953.238916] [] vzalloc_node+0x47/0x50 [44953.257089] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44953.280428] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44953.303192] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [44953.326532] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [44953.347868] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [44953.372903] [] kthread+0xcf/0xe0 [44953.389648] [] ? insert_kthread_work+0x40/0x40 [44953.410401] [] ret_from_fork+0x58/0x90 [44953.428869] [] ? insert_kthread_work+0x40/0x40 [44953.449626] Mem-Info: [44953.457792] active_anon:1 inactive_anon:8 isolated_anon:0 active_file:169 inactive_file:76 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9801 slab_unreclaimable:234973 mapped:1548 shmem:0 pagetables:1615 bounce:0 free:39275 free_pcp:1 free_cma:0 [44953.567994] Node 1 Normal free:45724kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:560kB inactive_file:252kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567992kB kernel_stack:18992kB pagetables:2932kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:775 all_unreclaimable? no [44953.723159] lowmem_reserve[]: 0 0 0 0 [44953.736116] Node 1 Normal: 1042*4kB (UEM) 725*8kB (UEM) 369*16kB (UEM) 341*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45728kB [44953.787403] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44953.816946] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44953.845616] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44953.875127] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44953.903753] 1807 total pagecache pages [44953.916885] 1 pages in swap cache [44953.928587] Swap cache stats: add 70092, delete 70091, find 14819/20062 [44953.951136] Free swap = 16238096kB [44953.963397] Total swap = 16319484kB [44953.975650] 8369063 pages RAM [44953.986171] 0 pages HighMem/MovableOnly [44953.999545] 241261 pages reserved [44954.011210] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44954.037803] [ 1003] 0 1003 24572 211 51 87 0 systemd-journal [44954.067535] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44954.094971] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44954.124105] [ 1081] 0 1081 1588718 4026 133 0 -1000 multipathd [44954.152417] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44954.179575] [ 2588] 81 2588 9199 178 21 138 -900 dbus-daemon [44954.208157] [ 2621] 0 2621 64917 325 78 276 0 sssd [44954.234748] [ 2622] 999 2622 136594 208 61 1634 0 polkitd [44954.262197] [ 2624] 0 2624 118083 228 83 772 0 NetworkManager [44954.291649] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44954.319086] [ 2637] 0 2637 5461 186 16 148 0 irqbalance [44954.347367] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44954.374505] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44954.402215] [ 2938] 0 2938 93409 328 124 621 0 sssd_be [44954.429644] [ 2940] 0 2940 66463 292 84 357 0 sssd_nss [44954.457346] [ 2941] 0 2941 61252 281 74 240 0 sssd_pam [44954.485036] [ 2942] 0 2942 59613 268 71 225 0 sssd_ssh [44954.512725] [ 2943] 0 2943 68163 268 86 338 0 sssd_pac [44954.540412] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44954.568094] [ 2955] 0 2955 6051 224 16 80 0 systemd-logind [44954.597485] [ 3689] 996 3689 24211 171 21 130 0 munged [44954.624596] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44954.651697] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44954.678224] [ 3859] 0 3859 22907 170 44 263 0 master [44954.705317] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44954.731826] [ 3883] 0 3883 6464 163 19 52 0 atd [44954.758059] [ 3900] 0 3900 151290 201 68 378 0 automount [44954.785910] [ 3922] 0 3922 31559 188 19 157 0 crond [44954.812591] [ 3971] 0 3971 27511 156 12 32 0 agetty [44954.839534] [ 3975] 0 3975 27511 160 10 33 0 agetty [44954.866489] [147960] 89 147960 25474 209 46 256 0 pickup [44954.893958] [151569] 0 151569 40498 272 78 312 0 sshd [44954.920851] [151570] 74 151570 27583 151 56 276 0 sshd [44954.947725] [151571] 0 151571 76 0 2 2 0 systemd-cgroups [44954.977728] [151572] 0 151572 76 0 2 2 0 systemd-cgroups [44955.007704] Out of memory: Kill process 2622 (polkitd) score 0 or sacrifice child [44955.032834] Killed process 2622 (polkitd) total-vm:546376kB, anon-rss:0kB, file-rss:832kB, shmem-rss:0kB [44955.371364] sssd_nss invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [44955.397418] sssd_nss cpuset=/ mems_allowed=0-1 [44955.412815] CPU: 6 PID: 2940 Comm: sssd_nss Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44955.449184] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44955.486996] ffff88040d668fd0 000000000e58b2fb ffff8804107bb920 ffffffff816a3e2d [44955.512272] ffff8804107bb9b0 ffffffff8169f246 ffffffff810e93ac ffff88042b790d70 [44955.537568] ffff88042b790d88 0000000000000206 0000000000000206 ffff8804107bb9a0 [44955.562871] Call Trace: [44955.571822] [] dump_stack+0x19/0x1b [44955.589675] [] dump_header+0x90/0x229 [44955.608082] [] ? ktime_get_ts64+0x4c/0xf0 [44955.627633] [] ? delayacct_end+0x8f/0xb0 [44955.646884] [] oom_kill_process+0x254/0x3d0 [44955.666992] [] ? oom_unkillable_task+0xcd/0x120 [44955.688242] [] ? find_lock_task_mm+0x56/0xc0 [44955.708635] [] out_of_memory+0x4b6/0x4f0 [44955.727872] [] __alloc_pages_slowpath+0x5d6/0x724 [44955.749706] [] __alloc_pages_nodemask+0x405/0x420 [44955.771525] [] alloc_pages_current+0x98/0x110 [44955.792206] [] __page_cache_alloc+0x97/0xb0 [44955.812303] [] filemap_fault+0x170/0x410 [44955.831483] [] ext4_filemap_fault+0x36/0x50 [ext4] [44955.853561] [] __do_fault+0x52/0xe0 [44955.871346] [] do_read_fault.isra.44+0x4b/0x130 [44955.892562] [] handle_mm_fault+0x691/0xfa0 [44955.912358] [] __do_page_fault+0x154/0x450 [44955.932158] [] do_page_fault+0x35/0x90 [44955.950823] [] page_fault+0x28/0x30 [44955.968603] Mem-Info: [44955.976944] active_anon:0 inactive_anon:11 isolated_anon:0 active_file:67 inactive_file:177 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9800 slab_unreclaimable:234968 mapped:1547 shmem:0 pagetables:1615 bounce:0 free:39334 free_pcp:0 free_cma:0 [44956.088566] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44956.228965] lowmem_reserve[]: 0 2580 15620 15620 [44956.245320] Node 0 DMA32 free:59484kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59756kB kernel_stack:1072kB pagetables:348kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:360 all_unreclaimable? yes [44956.395733] lowmem_reserve[]: 0 0 13040 13040 [44956.411409] Node 0 Normal free:36808kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:96kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16420kB slab_unreclaimable:312140kB kernel_stack:10784kB pagetables:3180kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:778 all_unreclaimable? yes [44956.567628] lowmem_reserve[]: 0 0 0 0 [44956.581026] Node 1 Normal free:45676kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:512kB inactive_file:300kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:568008kB kernel_stack:18976kB pagetables:2936kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1510 all_unreclaimable? yes [44956.738975] lowmem_reserve[]: 0 0 0 0 [44956.752500] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44956.798617] Node 0 DMA32: 470*4kB (UEM) 707*8kB (UEM) 506*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59504kB [44956.856366] Node 0 Normal: 184*4kB (UEM) 323*8kB (UEM) 596*16kB (UEM) 469*32kB (UEM) 120*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36824kB [44956.908543] Node 1 Normal: 1045*4kB (UEM) 725*8kB (UEM) 369*16kB (UEM) 338*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45644kB [44956.961369] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44956.991703] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44957.021121] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44957.051403] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44957.080813] 1783 total pagecache pages [44957.094818] 0 pages in swap cache [44957.107334] Swap cache stats: add 70176, delete 70176, find 14838/20095 [44957.130774] Free swap = 16238088kB [44957.143902] Total swap = 16319484kB [44957.157041] 8369063 pages RAM [44957.168413] 0 pages HighMem/MovableOnly [44957.182686] 241261 pages reserved [44957.195169] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44957.222604] [ 1003] 0 1003 24572 209 51 90 0 systemd-journal [44957.253178] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44957.281463] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44957.311526] [ 1081] 0 1081 1588718 4026 133 0 -1000 multipathd [44957.340698] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44957.368596] [ 2588] 81 2588 9199 177 21 139 -900 dbus-daemon [44957.398009] [ 2621] 0 2621 64917 325 78 278 0 sssd [44957.425391] [ 2643] 999 2622 136594 272 61 1633 0 gdbus [44957.452991] [ 2624] 0 2624 118083 228 83 773 0 NetworkManager [44957.483213] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44957.511361] [ 2637] 0 2637 5461 185 16 148 0 irqbalance [44957.540304] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44957.567992] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44957.596324] [ 2938] 0 2938 93409 328 124 622 0 sssd_be [44957.624294] [ 2940] 0 2940 66463 292 84 358 0 sssd_nss [44957.652557] [ 2941] 0 2941 61252 280 74 241 0 sssd_pam [44957.680799] [ 2942] 0 2942 59613 267 71 226 0 sssd_ssh [44957.708981] [ 2943] 0 2943 68163 267 86 339 0 sssd_pac [44957.737106] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44957.765210] [ 2955] 0 2955 6051 224 16 80 0 systemd-logind [44957.795047] [ 3689] 996 3689 24211 171 21 130 0 munged [44957.822589] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44957.850048] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44957.876936] [ 3859] 0 3859 22907 170 44 264 0 master [44957.904351] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44957.931125] [ 3883] 0 3883 6464 163 19 52 0 atd [44957.957644] [ 3900] 0 3900 151290 201 68 378 0 automount [44957.985855] [ 3922] 0 3922 31559 188 19 157 0 crond [44958.012878] [ 3971] 0 3971 27511 156 12 32 0 agetty [44958.040179] [ 3975] 0 3975 27511 160 10 33 0 agetty [44958.067450] [147960] 89 147960 25474 209 46 256 0 pickup [44958.095191] [151569] 0 151569 40498 272 78 312 0 sshd [44958.122391] [151570] 74 151570 27583 151 56 276 0 sshd [44958.149550] [151571] 0 151571 76 0 3 2 0 systemd-cgroups [44958.179832] [151572] 0 151572 79 0 3 4 0 systemd-cgroups [44958.210054] Out of memory: Kill process 2643 (gdbus) score 0 or sacrifice child [44958.234847] Killed process 2643 (gdbus) total-vm:546376kB, anon-rss:0kB, file-rss:1088kB, shmem-rss:0kB [44958.276632] gdbus: page allocation failure: order:0, mode:0x200da [44958.297434] CPU: 3 PID: 2643 Comm: gdbus Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44958.332714] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44958.370320] 00000000000200da 0000000081c1fb13 ffff8800b12575b8 ffffffff816a3e2d [44958.395475] ffff8800b1257648 ffffffff81188820 ffffffff8118b540 0000000000000000 [44958.420513] ffffffffffffffff 000200da00000000 ffff8800b1257648 0000000081c1fb13 [44958.445620] Call Trace: [44958.454258] [] dump_stack+0x19/0x1b [44958.468124] Lustre: 5694:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513255072/real 1513255072] req@ffff8803f87a0f00 x1586716054798672/t0(0) o400->soaked-MDT0002-lwp-OST0005@192.168.1.110@o2ib:12/10 lens 224/224 e 0 to 1 dl 1513255079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [44958.468126] Lustre: 5699:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513255072/real 1513255072] req@ffff8803f87a5700 x1586716054798752/t0(0) o400->soaked-MDT0001-lwp-OST0011@192.168.1.109@o2ib:12/10 lens 224/224 e 0 to 1 dl 1513255079 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [44958.468127] Lustre: 5694:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [44958.468128] Lustre: 5699:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [44958.468132] Lustre: soaked-MDT0002-lwp-OST000b: Connection to soaked-MDT0002 (at 192.168.1.110@o2ib) was lost; in progress operations using this service will wait for recovery to complete [44958.468134] Lustre: Skipped 5 previous similar messages [44958.805817] [] warn_alloc_failed+0x110/0x180 [44958.826159] [] ? drain_pages+0xb0/0xb0 [44958.844821] [] ? wake_up_atomic_t+0x30/0x30 [44958.864853] [] __alloc_pages_slowpath+0x6b6/0x724 [44958.886648] [] __alloc_pages_nodemask+0x405/0x420 [44958.908410] [] alloc_pages_vma+0xb5/0x200 [44958.927887] [] read_swap_cache_async+0xed/0x160 [44958.949016] [] swapin_readahead+0xa8/0x110 [44958.968719] [] handle_mm_fault+0xadb/0xfa0 [44958.988506] [] ? dequeue_task_fair+0x41e/0x660 [44959.009409] [] ? sched_clock_cpu+0x85/0xc0 [44959.029212] [] __do_page_fault+0x154/0x450 [44959.048934] [] do_page_fault+0x35/0x90 [44959.067528] [] ? schedule_hrtimeout_range_clock+0x12d/0x150 [44959.092142] [] page_fault+0x28/0x30 [44959.109940] [] ? do_sys_poll+0x184/0x580 [44959.129214] [] ? do_sys_poll+0x145/0x580 [44959.148426] [] ? sock_sendmsg+0xb0/0xf0 [44959.167366] [] ? sock_recvmsg+0xbf/0x100 [44959.186618] [] ? poll_select_copy_remaining+0x150/0x150 [44959.210062] [] ? poll_select_copy_remaining+0x150/0x150 [44959.233516] [] ? ___sys_sendmsg+0x3a9/0x3c0 [44959.253618] [] ? default_wake_function+0x12/0x20 [44959.275101] [] ? pollwake+0x73/0x90 [44959.292908] [] ? wake_up_state+0x20/0x20 [44959.312105] [] ? eventfd_ctx_read+0x67/0x1e0 [44959.332434] [] ? __sys_sendmsg+0x85/0x90 [44959.351639] [] SyS_poll+0x74/0x110 [44959.369150] [] ? __audit_syscall_exit+0x1e6/0x280 [44959.390862] [] system_call_fastpath+0x16/0x1b [44959.411508] Mem-Info: [44959.419799] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:169 inactive_file:149 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9800 slab_unreclaimable:234976 mapped:1552 shmem:0 pagetables:1617 bounce:0 free:38707 free_pcp:611 free_cma:0 [44959.531238] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44959.670442] lowmem_reserve[]: 0 2580 15620 15620 [44959.686390] Node 0 DMA32 free:59252kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:16kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59756kB kernel_stack:1072kB pagetables:352kB unstable:0kB bounce:0kB free_pcp:240kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:512 all_unreclaimable? yes [44959.836077] lowmem_reserve[]: 0 0 13040 13040 [44959.851311] Node 0 Normal free:36328kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:24kB inactive_file:128kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16420kB slab_unreclaimable:312140kB kernel_stack:10768kB pagetables:3180kB unstable:0kB bounce:0kB free_pcp:476kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:778 all_unreclaimable? yes [44960.006207] lowmem_reserve[]: 0 0 0 0 [44960.019222] Node 1 Normal free:43672kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:656kB inactive_file:452kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1840kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:568008kB kernel_stack:18976kB pagetables:2936kB unstable:0kB bounce:0kB free_pcp:1732kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:45104 all_unreclaimable? yes [44960.176381] lowmem_reserve[]: 0 0 0 0 [44960.189497] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44960.234781] Node 0 DMA32: 407*4kB (UEM) 707*8kB (UEM) 506*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59252kB [44960.291623] Node 0 Normal: 184*4kB (UEM) 323*8kB (UEM) 565*16kB (UEM) 469*32kB (UEM) 120*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36328kB [44960.342895] Node 1 Normal: 580*4kB (EM) 725*8kB (UEM) 368*16kB (EM) 335*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43672kB [44960.394090] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44960.423949] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44960.452917] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44960.482739] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44960.511789] 1864 total pagecache pages [44960.525339] 0 pages in swap cache [44960.537434] Swap cache stats: add 70234, delete 70234, find 14850/20121 [44960.560424] Free swap = 16238084kB [44960.573108] Total swap = 16319484kB [44960.585781] 8369063 pages RAM [44960.596708] 0 pages HighMem/MovableOnly [44960.610504] 241261 pages reserved [44961.348920] ll_ost_out01_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [44961.377083] ll_ost_out01_00 cpuset=/ mems_allowed=1 [44961.394360] CPU: 28 PID: 32074 Comm: ll_ost_out01_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44961.434801] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44961.472926] ffff88080801bf40 000000004282fc3a ffff8803c00ab998 ffffffff816a3e2d [44961.498551] ffff8803c00aba28 ffffffff8169f246 ffffffff810e93ac ffff8804183a1600 [44961.524242] ffff8804183a1618 0000000000000206 0000000000000206 ffff8803c00aba18 [44961.549877] Call Trace: [44961.559151] [] dump_stack+0x19/0x1b [44961.577302] [] dump_header+0x90/0x229 [44961.595912] [] ? ktime_get_ts64+0x4c/0xf0 [44961.615609] [] ? delayacct_end+0x8f/0xb0 [44961.635063] [] oom_kill_process+0x254/0x3d0 [44961.655371] [] ? oom_unkillable_task+0xcd/0x120 [44961.676719] [] ? find_lock_task_mm+0x56/0xc0 [44961.697200] [] out_of_memory+0x4b6/0x4f0 [44961.716498] [] __alloc_pages_slowpath+0x5d6/0x724 [44961.738359] [] __alloc_pages_nodemask+0x405/0x420 [44961.760188] [] __vmalloc_node_range+0x12f/0x270 [44961.781462] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44961.804987] [] vzalloc_node+0x47/0x50 [44961.823372] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44961.846890] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44961.869821] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [44961.893299] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [44961.914730] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [44961.939881] [] kthread+0xcf/0xe0 [44961.956721] [] ? insert_kthread_work+0x40/0x40 [44961.977547] [] ret_from_fork+0x58/0x90 [44961.996061] [] ? insert_kthread_work+0x40/0x40 [44962.016848] Mem-Info: [44962.025030] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:108 inactive_file:137 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9800 slab_unreclaimable:234975 mapped:1547 shmem:0 pagetables:1556 bounce:0 free:39367 free_pcp:0 free_cma:0 [44962.135238] Node 1 Normal free:45704kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:568008kB kernel_stack:18976kB pagetables:2936kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1305 all_unreclaimable? yes [44962.290015] lowmem_reserve[]: 0 0 0 0 [44962.302882] Node 1 Normal: 1046*4kB (UEM) 724*8kB (EM) 369*16kB (UEM) 342*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45768kB [44962.353689] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44962.383153] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44962.411738] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44962.441175] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44962.469727] 1804 total pagecache pages [44962.482814] 0 pages in swap cache [44962.494442] Swap cache stats: add 70295, delete 70295, find 14864/20150 [44962.516934] Free swap = 16244612kB [44962.529125] Total swap = 16319484kB [44962.541306] 8369063 pages RAM [44962.551760] 0 pages HighMem/MovableOnly [44962.565068] 241261 pages reserved [44962.576663] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44962.603168] [ 1003] 0 1003 24572 208 51 93 0 systemd-journal [44962.632820] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44962.660180] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44962.689256] [ 1081] 0 1081 1588718 4026 133 0 -1000 multipathd [44962.717485] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44962.744560] [ 2588] 81 2588 9199 177 21 140 -900 dbus-daemon [44962.773065] [ 2621] 0 2621 64917 324 78 278 0 sssd [44962.799571] [ 2624] 0 2624 118083 227 83 773 0 NetworkManager [44962.828938] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44962.856298] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44962.884515] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44962.911584] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44962.939218] [ 2938] 0 2938 93409 327 124 623 0 sssd_be [44962.966565] [ 2940] 0 2940 66463 291 84 360 0 sssd_nss [44962.994198] [ 2941] 0 2941 61252 279 74 243 0 sssd_pam [44963.021819] [ 2942] 0 2942 59613 265 71 227 0 sssd_ssh [44963.049434] [ 2943] 0 2943 68163 266 86 341 0 sssd_pac [44963.077043] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44963.104651] [ 2955] 0 2955 6051 223 16 82 0 systemd-logind [44963.133979] [ 3689] 996 3689 24211 171 21 130 0 munged [44963.161007] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44963.188032] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44963.214483] [ 3859] 0 3859 22907 168 44 266 0 master [44963.241500] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44963.267942] [ 3883] 0 3883 6464 163 19 52 0 atd [44963.294092] [ 3900] 0 3900 151290 201 68 378 0 automount [44963.321960] [ 3922] 0 3922 31559 188 19 157 0 crond [44963.348665] [ 3971] 0 3971 27511 156 12 32 0 agetty [44963.375632] [ 3975] 0 3975 27511 160 10 33 0 agetty [44963.402607] [147960] 89 147960 25474 209 46 256 0 pickup [44963.430100] [151569] 0 151569 40498 272 78 312 0 sshd [44963.456998] [151570] 74 151570 27583 151 56 276 0 sshd [44963.483892] [151571] 0 151571 79 0 3 6 0 systemd-cgroups [44963.513915] [151572] 0 151572 79 0 4 7 0 systemd-cgroups [44963.543913] Out of memory: Kill process 2624 (NetworkManager) score 0 or sacrifice child [44963.571045] Killed process 2624 (NetworkManager) total-vm:472332kB, anon-rss:0kB, file-rss:908kB, shmem-rss:0kB [44963.922380] gssproxy invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44963.948160] gssproxy cpuset=/ mems_allowed=0-1 [44963.963400] CPU: 2 PID: 2891 Comm: gssproxy Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44963.999495] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44964.037026] ffff8800b12d0fd0 00000000ab5f4d76 ffff88040d663988 ffffffff816a3e2d [44964.061989] ffff88040d663a18 ffffffff8169f246 ffffffff810e93ac ffff880410fb9670 [44964.087041] ffff880410fb9688 0000000000000206 0000000000000206 ffff88040d663a08 [44964.112071] Call Trace: [44964.120837] [] dump_stack+0x19/0x1b [44964.138439] [] dump_header+0x90/0x229 [44964.156607] [] ? ktime_get_ts64+0x4c/0xf0 [44964.175859] [] ? delayacct_end+0x8f/0xb0 [44964.194856] [] oom_kill_process+0x254/0x3d0 [44964.214773] [] ? oom_unkillable_task+0xcd/0x120 [44964.235779] [] ? find_lock_task_mm+0x56/0xc0 [44964.255888] [] out_of_memory+0x4b6/0x4f0 [44964.274893] [] __alloc_pages_slowpath+0x5d6/0x724 [44964.296485] [] __alloc_pages_nodemask+0x405/0x420 [44964.318106] [] alloc_pages_vma+0xb5/0x200 [44964.337372] [] ? sched_clock+0x9/0x10 [44964.355420] [] read_swap_cache_async+0xed/0x160 [44964.376489] [] swapin_readahead+0xa8/0x110 [44964.396030] [] handle_mm_fault+0xadb/0xfa0 [44964.415567] [] __do_page_fault+0x154/0x450 [44964.435164] [] do_page_fault+0x35/0x90 [44964.453593] [] page_fault+0x28/0x30 [44964.469021] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1513255079/real 1513255079] req@ffff8803f57b7800 x1586716054798912/t0(0) o38->soaked-MDT0002-lwp-OST0011@192.168.1.110@o2ib:12/10 lens 520/544 e 0 to 1 dl 1513255085 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [44964.469022] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [44964.600348] Mem-Info: [44964.608567] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:116 inactive_file:144 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9800 slab_unreclaimable:234969 mapped:1552 shmem:0 pagetables:1475 bounce:0 free:39236 free_pcp:163 free_cma:0 [44964.720059] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44964.859676] lowmem_reserve[]: 0 2580 15620 15620 [44964.875861] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:20kB inactive_file:0kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:584kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59756kB kernel_stack:1088kB pagetables:268kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2486 all_unreclaimable? yes [44965.026671] lowmem_reserve[]: 0 0 13040 13040 [44965.042047] Node 0 Normal free:36348kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:124kB inactive_file:84kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3800kB shmem:0kB slab_reclaimable:16420kB slab_unreclaimable:312136kB kernel_stack:10720kB pagetables:2720kB unstable:0kB bounce:0kB free_pcp:536kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:543 all_unreclaimable? yes [44965.197884] lowmem_reserve[]: 0 0 0 0 [44965.211038] Node 1 Normal free:45584kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1824kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567984kB kernel_stack:18976kB pagetables:2912kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1513 all_unreclaimable? yes [44965.367108] lowmem_reserve[]: 0 0 0 0 [44965.380228] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44965.425536] Node 0 DMA32: 440*4kB (UEM) 712*8kB (UEM) 507*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59440kB [44965.482419] Node 0 Normal: 90*4kB (EM) 323*8kB (UEM) 576*16kB (UEM) 470*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36224kB [44965.533044] Node 1 Normal: 1049*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 338*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45684kB [44965.584894] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44965.614740] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44965.643733] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44965.673567] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44965.702536] 1810 total pagecache pages [44965.716074] 1 pages in swap cache [44965.728168] Swap cache stats: add 70341, delete 70340, find 14870/20167 [44965.751126] Free swap = 16247592kB [44965.763759] Total swap = 16319484kB [44965.776373] 8369063 pages RAM [44965.787245] 0 pages HighMem/MovableOnly [44965.800963] 241261 pages reserved [44965.812958] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44965.839857] [ 1003] 0 1003 24572 207 51 93 0 systemd-journal [44965.869905] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44965.897659] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44965.927129] [ 1081] 0 1081 1588718 4026 133 0 -1000 multipathd [44965.955753] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44965.983209] [ 2588] 81 2588 9199 176 21 140 -900 dbus-daemon [44966.012065] [ 2621] 0 2621 64917 324 78 279 0 sssd [44966.038897] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44966.066549] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44966.095061] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44966.122408] [ 2891] 0 2891 67755 210 45 676 0 gssproxy [44966.150306] [ 2938] 0 2938 93409 327 124 623 0 sssd_be [44966.177895] [ 2940] 0 2940 66463 291 84 360 0 sssd_nss [44966.205764] [ 2941] 0 2941 61252 279 74 243 0 sssd_pam [44966.233593] [ 2942] 0 2942 59613 265 71 227 0 sssd_ssh [44966.261412] [ 2943] 0 2943 68163 266 86 341 0 sssd_pac [44966.289192] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44966.316953] [ 2955] 0 2955 6051 222 16 82 0 systemd-logind [44966.346408] [ 3689] 996 3689 24211 171 21 130 0 munged [44966.373552] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44966.400665] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44966.427212] [ 3859] 0 3859 22907 167 44 266 0 master [44966.454309] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44966.480813] [ 3883] 0 3883 6464 163 19 52 0 atd [44966.507003] [ 3900] 0 3900 151290 201 68 378 0 automount [44966.534887] [ 3922] 0 3922 31559 188 19 157 0 crond [44966.561627] [ 3971] 0 3971 27511 156 12 32 0 agetty [44966.588631] [ 3975] 0 3975 27511 160 10 33 0 agetty [44966.615655] [147960] 89 147960 25474 209 46 256 0 pickup [44966.643188] [151569] 0 151569 40498 272 78 312 0 sshd [44966.670128] [151570] 74 151570 27583 151 56 276 0 sshd [44966.697043] [151571] 0 151571 79 0 4 6 0 systemd-cgroups [44966.727085] [151572] 0 151572 79 0 4 7 0 systemd-cgroups [44966.757101] Out of memory: Kill process 2938 (sssd_be) score 0 or sacrifice child [44966.782241] Killed process 2938 (sssd_be) total-vm:373636kB, anon-rss:0kB, file-rss:1308kB, shmem-rss:0kB [44966.917914] irqbalance invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44966.944301] irqbalance cpuset=/ mems_allowed=0-1 [44966.960098] CPU: 7 PID: 2637 Comm: irqbalance Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44966.996821] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44967.034403] ffff8808295c8fd0 00000000c9699953 ffff88040f34f988 ffffffff816a3e2d [44967.059468] ffff88040f34fa18 ffffffff8169f246 ffffffff810e93ac ffff88042b097de0 [44967.084527] ffff88042b097df8 0000000000000202 0000000000000202 ffff88040f34fa08 [44967.109590] Call Trace: [44967.118296] [] dump_stack+0x19/0x1b [44967.135876] [] dump_header+0x90/0x229 [44967.154013] [] ? ktime_get_ts64+0x4c/0xf0 [44967.173300] [] ? delayacct_end+0x8f/0xb0 [44967.192286] [] oom_kill_process+0x254/0x3d0 [44967.212119] [] ? oom_unkillable_task+0xcd/0x120 [44967.233114] [] ? find_lock_task_mm+0x56/0xc0 [44967.253256] [] out_of_memory+0x4b6/0x4f0 [44967.272230] [] __alloc_pages_slowpath+0x5d6/0x724 [44967.293797] [] __alloc_pages_nodemask+0x405/0x420 [44967.315351] [] alloc_pages_vma+0xb5/0x200 [44967.334588] [] ? end_swap_bio_write+0x80/0x80 [44967.355010] [] read_swap_cache_async+0xed/0x160 [44967.376002] [] ? mem_cgroup_update_page_stat+0x16/0x50 [44967.399009] [] swapin_readahead+0xa8/0x110 [44967.418587] [] handle_mm_fault+0xadb/0xfa0 [44967.438130] [] __do_page_fault+0x154/0x450 [44967.457701] [] do_page_fault+0x35/0x90 [44967.476127] [] page_fault+0x28/0x30 [44967.493684] Mem-Info: [44967.501813] active_anon:0 inactive_anon:5 isolated_anon:0 active_file:124 inactive_file:121 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9792 slab_unreclaimable:234969 mapped:1548 shmem:0 pagetables:1352 bounce:0 free:39413 free_pcp:89 free_cma:0 [44967.612410] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44967.751722] lowmem_reserve[]: 0 2580 15620 15620 [44967.767768] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:16kB active_file:0kB inactive_file:24kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59756kB kernel_stack:1088kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:337 all_unreclaimable? yes [44967.918132] lowmem_reserve[]: 0 0 13040 13040 [44967.933450] Node 0 Normal free:36956kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:184kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312136kB kernel_stack:10720kB pagetables:2280kB unstable:0kB bounce:0kB free_pcp:244kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:566 all_unreclaimable? yes [44968.088651] lowmem_reserve[]: 0 0 0 0 [44968.101754] Node 1 Normal free:45684kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1824kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567984kB kernel_stack:18976kB pagetables:2912kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1427 all_unreclaimable? yes [44968.257981] lowmem_reserve[]: 0 0 0 0 [44968.271124] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44968.316525] Node 0 DMA32: 438*4kB (UEM) 712*8kB (UEM) 508*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59448kB [44968.373505] Node 0 Normal: 150*4kB (EM) 329*8kB (UEM) 592*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36832kB [44968.424510] Node 1 Normal: 1049*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 338*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45684kB [44968.476475] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44968.506367] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44968.535399] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44968.565284] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44968.594290] 1799 total pagecache pages [44968.607851] 6 pages in swap cache [44968.619990] Swap cache stats: add 70386, delete 70380, find 14873/20173 [44968.642996] Free swap = 16250068kB [44968.655685] Total swap = 16319484kB [44968.668396] 8369063 pages RAM [44968.679387] 0 pages HighMem/MovableOnly [44968.693223] 241261 pages reserved [44968.705344] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44968.732385] [ 1003] 0 1003 24572 207 51 93 0 systemd-journal [44968.762565] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44968.790451] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44968.820033] [ 1081] 0 1081 1588718 4026 133 0 -1000 multipathd [44968.848790] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44968.876353] [ 2588] 81 2588 9199 176 21 140 -900 dbus-daemon [44968.905331] [ 2621] 0 2621 64917 322 78 279 0 sssd [44968.932277] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44968.960048] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44968.988660] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44969.016107] [ 2891] 0 2891 67755 210 45 678 0 gssproxy [44969.044132] [ 2940] 0 2940 66463 291 84 360 0 sssd_nss [44969.072130] [ 2941] 0 2941 61252 279 74 243 0 sssd_pam [44969.100093] [ 2942] 0 2942 59613 265 71 227 0 sssd_ssh [44969.127986] [ 2943] 0 2943 68163 265 86 342 0 sssd_pac [44969.155894] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44969.183740] [ 2955] 0 2955 6051 222 16 82 0 systemd-logind [44969.213292] [ 3689] 996 3689 24211 171 21 130 0 munged [44969.240531] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44969.267766] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44969.294434] [ 3859] 0 3859 22907 167 44 266 0 master [44969.321622] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44969.348232] [ 3883] 0 3883 6464 163 19 52 0 atd [44969.374533] [ 3900] 0 3900 151290 201 68 378 0 automount [44969.402516] [ 3922] 0 3922 31559 187 19 157 0 crond [44969.429344] [ 3971] 0 3971 27511 156 12 32 0 agetty [44969.456448] [ 3975] 0 3975 27511 160 10 33 0 agetty [44969.483570] [147960] 89 147960 25474 209 46 256 0 pickup [44969.511183] [151569] 0 151569 40498 272 78 312 0 sshd [44969.538200] [151570] 74 151570 27583 151 56 276 0 sshd [44969.565208] [151571] 0 151571 79 0 4 6 0 systemd-cgroups [44969.595356] [151572] 0 151572 79 0 4 7 0 systemd-cgroups [44969.625446] [151574] 0 151574 1 0 1 1 0 systemd-cgroups [44969.655503] Out of memory: Kill process 2891 (gssproxy) score 0 or sacrifice child [44969.680984] Killed process 2891 (gssproxy) total-vm:271020kB, anon-rss:0kB, file-rss:840kB, shmem-rss:0kB [44970.510390] systemd-cgroups invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [44970.538193] systemd-cgroups cpuset=/ mems_allowed=0-1 [44970.555419] CPU: 7 PID: 151572 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44970.594747] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44970.632339] ffff880811c78fd0 0000000050754271 ffff8808243a3908 ffffffff816a3e2d [44970.657420] ffff8808243a3998 ffffffff8169f246 ffffffff810e93ac ffff8807d02399f0 [44970.682493] ffff8807d0239a08 0000000000000206 0000000000000206 ffff8808243a3988 [44970.707580] Call Trace: [44970.716304] [] dump_stack+0x19/0x1b [44970.733902] [] dump_header+0x90/0x229 [44970.752082] [] ? ktime_get_ts64+0x4c/0xf0 [44970.771379] [] ? delayacct_end+0x8f/0xb0 [44970.790394] [] oom_kill_process+0x254/0x3d0 [44970.810251] [] ? oom_unkillable_task+0xcd/0x120 [44970.831250] [] ? find_lock_task_mm+0x56/0xc0 [44970.851394] [] out_of_memory+0x4b6/0x4f0 [44970.870387] [] __alloc_pages_slowpath+0x5d6/0x724 [44970.891981] [] __alloc_pages_nodemask+0x405/0x420 [44970.913552] [] alloc_pages_current+0x98/0x110 [44970.933992] [] __page_cache_alloc+0x97/0xb0 [44970.953856] [] filemap_fault+0x170/0x410 [44970.972891] [] ext4_filemap_fault+0x36/0x50 [ext4] [44970.994758] [] __do_fault+0x52/0xe0 [44971.012343] [] do_cow_fault+0xab/0x290 [44971.030787] [] handle_mm_fault+0x2bd/0xfa0 [44971.050368] [] ? change_protection+0x65/0xa0 [44971.070537] [] __do_page_fault+0x154/0x450 [44971.090139] [] do_page_fault+0x35/0x90 [44971.108594] [] page_fault+0x28/0x30 [44971.126187] Mem-Info: [44971.134335] active_anon:0 inactive_anon:3 isolated_anon:0 active_file:311 inactive_file:0 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9792 slab_unreclaimable:234977 mapped:1548 shmem:0 pagetables:1352 bounce:0 free:39357 free_pcp:0 free_cma:0 [44971.244183] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44971.383588] lowmem_reserve[]: 0 2580 15620 15620 [44971.399680] Node 0 DMA32 free:59404kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59788kB kernel_stack:1104kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:566 all_unreclaimable? yes [44971.549539] lowmem_reserve[]: 0 0 13040 13040 [44971.564826] Node 0 Normal free:36856kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:168kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312136kB kernel_stack:10720kB pagetables:2140kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:692 all_unreclaimable? yes [44971.719269] lowmem_reserve[]: 0 0 0 0 [44971.732293] Node 1 Normal free:44604kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:544kB inactive_file:508kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1832kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567968kB kernel_stack:18880kB pagetables:2872kB unstable:0kB bounce:0kB free_pcp:728kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1477 all_unreclaimable? yes [44971.889001] lowmem_reserve[]: 0 0 0 0 [44971.902110] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44971.947464] Node 0 DMA32: 439*4kB (UEM) 712*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59404kB [44972.004426] Node 0 Normal: 206*4kB (UEM) 331*8kB (UEM) 586*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36976kB [44972.055684] Node 1 Normal: 829*4kB (UEM) 725*8kB (UEM) 368*16kB (EM) 333*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44604kB [44972.107032] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44972.136913] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44972.165928] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44972.195790] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44972.224782] 1855 total pagecache pages [44972.238332] 0 pages in swap cache [44972.250451] Swap cache stats: add 70463, delete 70463, find 14887/20197 [44972.273448] Free swap = 16252764kB [44972.286148] Total swap = 16319484kB [44972.298859] 8369063 pages RAM [44972.309844] 0 pages HighMem/MovableOnly [44972.323693] 241261 pages reserved [44972.335803] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44972.362827] [ 1003] 0 1003 24572 207 51 93 0 systemd-journal [44972.392988] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44972.420862] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44972.450454] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [44972.479211] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44972.506781] [ 2588] 81 2588 9199 176 21 140 -900 dbus-daemon [44972.535771] [ 2621] 0 2621 64917 322 78 279 0 sssd [44972.562690] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44972.590468] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44972.619091] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44972.646549] [ 2940] 0 2940 66463 290 84 360 0 sssd_nss [44972.674563] [ 2941] 0 2941 61252 278 74 245 0 sssd_pam [44972.702557] [ 2942] 0 2942 59613 265 71 227 0 sssd_ssh [44972.730521] [ 2943] 0 2943 68163 264 86 343 0 sssd_pac [44972.758456] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44972.786373] [ 2955] 0 2955 6051 221 16 83 0 systemd-logind [44972.815959] [ 3689] 996 3689 24211 171 21 130 0 munged [44972.843234] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44972.870489] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44972.897150] [ 3859] 0 3859 22907 167 44 266 0 master [44972.924371] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44972.951009] [ 3883] 0 3883 6464 163 19 52 0 atd [44972.977337] [ 3900] 0 3900 151290 201 68 378 0 automount [44973.005356] [ 3922] 0 3922 31559 187 19 159 0 crond [44973.032204] [ 3971] 0 3971 27511 156 12 32 0 agetty [44973.059318] [ 3975] 0 3975 27511 160 10 33 0 agetty [44973.086454] [147960] 89 147960 25474 209 46 256 0 pickup [44973.114118] [151569] 0 151569 40498 272 78 312 0 sshd [44973.141161] [151570] 74 151570 27583 151 56 276 0 sshd [44973.168202] [151571] 0 151571 79 0 4 7 0 systemd-cgroups [44973.198337] [151572] 0 151572 79 0 4 7 0 systemd-cgroups [44973.228476] [151574] 0 151574 1 0 1 1 0 systemd-cgroups [44973.258561] Out of memory: Kill process 2940 (sssd_nss) score 0 or sacrifice child [44973.284069] Killed process 2940 (sssd_nss) total-vm:265852kB, anon-rss:0kB, file-rss:1160kB, shmem-rss:0kB [44973.338379] systemd-logind invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44973.366244] systemd-logind cpuset=/ mems_allowed=0-1 [44973.383160] CPU: 25 PID: 2955 Comm: systemd-logind Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44973.422003] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44973.459611] ffff88041876bf40 0000000068fac475 ffff88042ce7f988 ffffffff816a3e2d [44973.484689] ffff88042ce7fa18 ffffffff8169f246 ffffffff810e93ac ffff880036accf30 [44973.509719] ffff880036accf48 0000000000000202 0000000000000202 ffff88042ce7fa08 [44973.534777] Call Trace: [44973.543450] [] dump_stack+0x19/0x1b [44973.561092] [] dump_header+0x90/0x229 [44973.579522] [] ? ktime_get_ts64+0x4c/0xf0 [44973.598855] [] ? delayacct_end+0x8f/0xb0 [44973.617830] [] oom_kill_process+0x254/0x3d0 [44973.637716] [] ? oom_unkillable_task+0xcd/0x120 [44973.658666] [] ? find_lock_task_mm+0x56/0xc0 [44973.678754] [] out_of_memory+0x4b6/0x4f0 [44973.697699] [] __alloc_pages_slowpath+0x5d6/0x724 [44973.719219] [] __alloc_pages_nodemask+0x405/0x420 [44973.740736] [] alloc_pages_vma+0xb5/0x200 [44973.759964] [] read_swap_cache_async+0xed/0x160 [44973.780910] [] ? mem_cgroup_update_page_stat+0x16/0x50 [44973.803862] [] swapin_readahead+0xa8/0x110 [44973.823384] [] handle_mm_fault+0xadb/0xfa0 [44973.842906] [] __do_page_fault+0x154/0x450 [44973.862428] [] do_page_fault+0x35/0x90 [44973.880809] [] page_fault+0x28/0x30 [44973.898333] Mem-Info: [44973.906389] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:96 inactive_file:108 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9792 slab_unreclaimable:234973 mapped:1574 shmem:0 pagetables:1307 bounce:0 free:39127 free_pcp:0 free_cma:0 [44974.016477] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44974.155544] lowmem_reserve[]: 0 2580 15620 15620 [44974.171547] Node 0 DMA32 free:59476kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59788kB kernel_stack:1104kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2641 all_unreclaimable? no [44974.320486] lowmem_reserve[]: 0 0 13040 13040 [44974.335742] Node 0 Normal free:36976kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:28kB inactive_file:124kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312136kB kernel_stack:10720kB pagetables:2140kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:692 all_unreclaimable? yes [44974.490315] lowmem_reserve[]: 0 0 0 0 [44974.503326] Node 1 Normal free:45576kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567968kB kernel_stack:18880kB pagetables:2536kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1303 all_unreclaimable? yes [44974.659151] lowmem_reserve[]: 0 0 0 0 [44974.672187] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44974.717338] Node 0 DMA32: 465*4kB (UEM) 712*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59508kB [44974.774078] Node 0 Normal: 206*4kB (UEM) 331*8kB (UEM) 586*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36976kB [44974.825127] Node 1 Normal: 1043*4kB (EM) 725*8kB (UEM) 374*16kB (UEM) 337*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45684kB [44974.876554] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44974.906319] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44974.935234] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44974.964986] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44974.993874] 1790 total pagecache pages [44975.007308] 1 pages in swap cache [44975.019308] Swap cache stats: add 70492, delete 70491, find 14894/20210 [44975.042195] Free swap = 16254180kB [44975.054792] Total swap = 16319484kB [44975.067378] 8369063 pages RAM [44975.078244] 0 pages HighMem/MovableOnly [44975.091966] 241261 pages reserved [44975.103968] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44975.130979] [ 1003] 0 1003 24572 207 51 93 0 systemd-journal [44975.161039] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44975.188799] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44975.218280] [ 1081] 0 1081 1588718 3982 133 0 -1000 multipathd [44975.246910] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44975.274371] [ 2588] 81 2588 9199 176 21 140 -900 dbus-daemon [44975.303244] [ 2621] 0 2621 64917 322 78 279 0 sssd [44975.330083] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44975.357756] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44975.386275] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44975.413642] [ 2941] 0 2941 61252 276 74 248 0 sssd_pam [44975.441545] [ 2942] 0 2942 59613 265 71 228 0 sssd_ssh [44975.469437] [ 2943] 0 2943 68163 263 86 343 0 sssd_pac [44975.497291] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44975.525122] [ 2955] 0 2955 6051 221 16 83 0 systemd-logind [44975.554662] [ 3689] 996 3689 24211 171 21 130 0 munged [44975.581878] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44975.609073] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44975.635670] [ 3859] 0 3859 22907 167 44 267 0 master [44975.662832] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44975.689406] [ 3883] 0 3883 6464 163 19 52 0 atd [44975.715670] [ 3900] 0 3900 151290 201 68 378 0 automount [44975.743630] [ 3922] 0 3922 31559 186 19 159 0 crond [44975.770424] [ 3971] 0 3971 27511 156 12 32 0 agetty [44975.797475] [ 3975] 0 3975 27511 160 10 33 0 agetty [44975.824546] [147960] 89 147960 25474 209 46 256 0 pickup [44975.852149] [151569] 0 151569 40498 272 78 312 0 sshd [44975.879159] [151570] 74 151570 27583 151 56 276 0 sshd [44975.906149] [151571] 0 151571 79 0 4 7 0 systemd-cgroups [44975.936267] [151572] 0 151572 79 0 4 8 0 systemd-cgroups [44975.966358] [151574] 0 151574 41 0 1 1 0 systemd-cgroups [44975.996416] Out of memory: Kill process 2943 (sssd_pac) score 0 or sacrifice child [44976.021888] Killed process 2943 (sssd_pac) total-vm:272652kB, anon-rss:0kB, file-rss:1052kB, shmem-rss:0kB [44976.311978] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44976.340071] systemd-cgroups cpuset=/ mems_allowed=0-1 [44976.357545] CPU: 19 PID: 151574 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44976.397042] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44976.434593] ffff88080403af70 000000000d7b818d ffff8800b1257a00 ffffffff816a3e2d [44976.459638] ffff8800b1257a90 ffffffff8169f246 ffffffff810e93ac ffff880370f99ec0 [44976.484743] ffff880370f99ed8 0000000000000206 0000000000000206 ffff8800b1257a80 [44976.509784] Call Trace: [44976.518520] [] dump_stack+0x19/0x1b [44976.536172] [] dump_header+0x90/0x229 [44976.554299] [] ? ktime_get_ts64+0x4c/0xf0 [44976.573691] [] ? delayacct_end+0x8f/0xb0 [44976.592777] [] oom_kill_process+0x254/0x3d0 [44976.612681] [] ? oom_unkillable_task+0xcd/0x120 [44976.633754] [] ? find_lock_task_mm+0x56/0xc0 [44976.653929] [] out_of_memory+0x4b6/0x4f0 [44976.672965] [] __alloc_pages_slowpath+0x5d6/0x724 [44976.694491] [] __alloc_pages_nodemask+0x405/0x420 [44976.716060] [] alloc_pages_vma+0xb5/0x200 [44976.735366] [] ? anon_vma_interval_tree_insert+0x8b/0x90 [44976.758939] [] do_cow_fault+0x66/0x290 [44976.777335] [] handle_mm_fault+0x2bd/0xfa0 [44976.796929] [] __do_page_fault+0x154/0x450 [44976.816513] [] do_page_fault+0x35/0x90 [44976.835050] [] page_fault+0x28/0x30 [44976.852661] Mem-Info: [44976.860776] active_anon:0 inactive_anon:9 isolated_anon:0 active_file:89 inactive_file:158 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9792 slab_unreclaimable:234971 mapped:1548 shmem:0 pagetables:1140 bounce:0 free:39411 free_pcp:6 free_cma:0 [44976.971036] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44977.110377] lowmem_reserve[]: 0 2580 15620 15620 [44977.126501] Node 0 DMA32 free:59372kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59788kB kernel_stack:1104kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:484 all_unreclaimable? yes [44977.275834] lowmem_reserve[]: 0 0 13040 13040 [44977.291174] Node 0 Normal free:36988kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:36kB active_file:44kB inactive_file:148kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3796kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312136kB kernel_stack:10720kB pagetables:1808kB unstable:0kB bounce:0kB free_pcp:28kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:786 all_unreclaimable? yes [44977.446671] lowmem_reserve[]: 0 0 0 0 [44977.459815] Node 1 Normal free:45708kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567960kB kernel_stack:18864kB pagetables:2536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1363 all_unreclaimable? yes [44977.616227] lowmem_reserve[]: 0 0 0 0 [44977.629414] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44977.674761] Node 0 DMA32: 431*4kB (UEM) 712*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59372kB [44977.731810] Node 0 Normal: 249*4kB (UEM) 323*8kB (UEM) 580*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36988kB [44977.783074] Node 1 Normal: 1043*4kB (EM) 726*8kB (UEM) 375*16kB (UEM) 337*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45708kB [44977.834811] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44977.864661] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44977.893695] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44977.923542] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44977.952515] 1803 total pagecache pages [44977.966063] 3 pages in swap cache [44977.978177] Swap cache stats: add 70520, delete 70517, find 14903/20226 [44978.001218] Free swap = 16255420kB [44978.014004] Total swap = 16319484kB [44978.026725] 8369063 pages RAM [44978.037637] 0 pages HighMem/MovableOnly [44978.051442] 241261 pages reserved [44978.063543] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44978.090618] [ 1003] 0 1003 24572 207 51 93 0 systemd-journal [44978.120782] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44978.148685] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44978.178300] [ 1081] 0 1081 1588718 3982 133 0 -1000 multipathd [44978.206974] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44978.234580] [ 2588] 81 2588 9199 175 21 141 -900 dbus-daemon [44978.263539] [ 2621] 0 2621 64917 322 78 279 0 sssd [44978.290565] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44978.318337] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44978.347062] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44978.374529] [ 2941] 0 2941 61252 276 74 248 0 sssd_pam [44978.402547] [ 2942] 0 2942 59613 264 71 228 0 sssd_ssh [44978.430488] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44978.458418] [ 2955] 0 2955 6051 221 16 83 0 systemd-logind [44978.488069] [ 3689] 996 3689 24211 171 21 130 0 munged [44978.515428] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44978.542734] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44978.569458] [ 3859] 0 3859 22907 166 44 267 0 master [44978.596773] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44978.623387] [ 3883] 0 3883 6464 163 19 52 0 atd [44978.649689] [ 3900] 0 3900 151290 201 68 378 0 automount [44978.677702] [ 3922] 0 3922 31559 185 19 160 0 crond [44978.704586] [ 3971] 0 3971 27511 156 12 32 0 agetty [44978.731699] [ 3975] 0 3975 27511 160 10 33 0 agetty [44978.758885] [147960] 89 147960 25474 209 46 256 0 pickup [44978.786555] [151569] 0 151569 40498 272 78 312 0 sshd [44978.813592] [151570] 74 151570 27583 151 56 276 0 sshd [44978.840656] [151571] 0 151571 79 0 4 7 0 systemd-cgroups [44978.870863] [151572] 0 151572 79 0 4 8 0 systemd-cgroups [44978.901052] [151574] 0 151574 610 27 4 10 0 systemd-cgroups [44978.931236] Out of memory: Kill process 2621 (sssd) score 0 or sacrifice child [44978.955626] Killed process 2941 (sssd_pam) total-vm:245008kB, anon-rss:0kB, file-rss:1104kB, shmem-rss:0kB [44978.994909] sssd_ssh invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44979.020725] sssd_ssh cpuset=/ mems_allowed=0-1 [44979.035911] CPU: 20 PID: 2942 Comm: sssd_ssh Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44979.072294] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44979.109893] ffff88040d66cf10 00000000b0166cfe ffff880417ac3988 ffffffff816a3e2d [44979.134906] ffff880417ac3a18 ffffffff8169f246 ffffffff810e93ac ffff88042b790ad0 [44979.159960] ffff88042b790ae8 0000000000000202 ffff880417ac3a70 ffff880417ac3a08 [44979.185157] Call Trace: [44979.193847] [] dump_stack+0x19/0x1b [44979.211394] [] dump_header+0x90/0x229 [44979.229498] [] ? ktime_get_ts64+0x4c/0xf0 [44979.248759] [] oom_kill_process+0x254/0x3d0 [44979.268605] [] ? oom_unkillable_task+0xcd/0x120 [44979.289591] [] ? find_lock_task_mm+0x56/0xc0 [44979.309685] [] out_of_memory+0x4b6/0x4f0 [44979.328651] [] __alloc_pages_slowpath+0x5d6/0x724 [44979.350419] [] __alloc_pages_nodemask+0x405/0x420 [44979.371949] [] alloc_pages_vma+0xb5/0x200 [44979.391191] [] read_swap_cache_async+0xed/0x160 [44979.412166] [] ? mem_cgroup_update_page_stat+0x16/0x50 [44979.435127] [] swapin_readahead+0xa8/0x110 [44979.454640] [] handle_mm_fault+0xadb/0xfa0 [44979.474183] [] __do_page_fault+0x154/0x450 [44979.493727] [] do_page_fault+0x35/0x90 [44979.512128] [] page_fault+0x28/0x30 [44979.529831] Mem-Info: [44979.538126] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:94 inactive_file:162 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9792 slab_unreclaimable:234971 mapped:1548 shmem:0 pagetables:1066 bounce:0 free:39349 free_pcp:151 free_cma:0 [44979.648724] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44979.787820] lowmem_reserve[]: 0 2580 15620 15620 [44979.803841] Node 0 DMA32 free:59372kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59788kB kernel_stack:1104kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1836 all_unreclaimable? yes [44979.953356] lowmem_reserve[]: 0 0 13040 13040 [44979.968626] Node 0 Normal free:36740kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:48kB inactive_file:148kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3796kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312136kB kernel_stack:10720kB pagetables:1808kB unstable:0kB bounce:0kB free_pcp:284kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1488 all_unreclaimable? yes [44980.124376] lowmem_reserve[]: 0 0 0 0 [44980.137476] Node 1 Normal free:45708kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567960kB kernel_stack:18864kB pagetables:2240kB unstable:0kB bounce:0kB free_pcp:324kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1363 all_unreclaimable? yes [44980.294414] lowmem_reserve[]: 0 0 0 0 [44980.307524] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44980.353009] Node 0 DMA32: 431*4kB (UEM) 712*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59372kB [44980.410105] Node 0 Normal: 187*4kB (UEM) 323*8kB (UEM) 580*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36740kB [44980.461434] Node 1 Normal: 1043*4kB (EM) 726*8kB (UEM) 375*16kB (UEM) 337*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45708kB [44980.513157] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44980.543208] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44980.572140] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44980.601961] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44980.630924] 1801 total pagecache pages [44980.644408] 0 pages in swap cache [44980.656702] Swap cache stats: add 70521, delete 70521, find 14904/20228 [44980.679696] Free swap = 16256332kB [44980.692478] Total swap = 16319484kB [44980.705343] 8369063 pages RAM [44980.716264] 0 pages HighMem/MovableOnly [44980.730027] 241261 pages reserved [44980.742072] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44980.769258] [ 1003] 0 1003 24572 207 51 93 0 systemd-journal [44980.799381] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44980.827322] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44980.857048] [ 1081] 0 1081 1588718 3982 133 0 -1000 multipathd [44980.885681] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44980.913391] [ 2588] 81 2588 9199 175 21 141 -900 dbus-daemon [44980.942296] [ 2621] 0 2621 64917 322 78 279 0 sssd [44980.969378] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44980.997100] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44981.025672] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44981.053075] [ 2942] 0 2942 59613 264 71 228 0 sssd_ssh [44981.081183] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44981.109092] [ 2955] 0 2955 6051 221 16 83 0 systemd-logind [44981.138695] [ 3689] 996 3689 24211 171 21 130 0 munged [44981.166127] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44981.193374] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44981.220080] [ 3859] 0 3859 22907 166 44 267 0 master [44981.247343] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44981.273984] [ 3883] 0 3883 6464 163 19 52 0 atd [44981.300307] [ 3900] 0 3900 151290 201 68 378 0 automount [44981.328325] [ 3922] 0 3922 31559 185 19 160 0 crond [44981.355380] [ 3971] 0 3971 27511 156 12 32 0 agetty [44981.382494] [ 3975] 0 3975 27511 160 10 33 0 agetty [44981.409846] [147960] 89 147960 25474 209 46 256 0 pickup [44981.437477] [151569] 0 151569 40498 272 78 312 0 sshd [44981.464533] [151570] 74 151570 27583 151 56 276 0 sshd [44981.491590] [151571] 0 151571 79 0 4 7 0 systemd-cgroups [44981.521836] [151572] 0 151572 79 0 4 8 0 systemd-cgroups [44981.552100] [151574] 0 151574 610 27 4 10 0 systemd-cgroups [44981.582235] Out of memory: Kill process 2621 (sssd) score 0 or sacrifice child [44981.606768] Killed process 2942 (sssd_ssh) total-vm:238452kB, anon-rss:0kB, file-rss:1056kB, shmem-rss:0kB [44981.640376] systemd-logind invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44981.668671] systemd-logind cpuset=/ mems_allowed=0-1 [44981.685968] CPU: 27 PID: 2955 Comm: systemd-logind Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44981.724655] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44981.762208] ffff88041876bf40 0000000068fac475 ffff88042ce7f988 ffffffff816a3e2d [44981.787241] ffff88042ce7fa18 ffffffff8169f246 ffffffff810e93ac ffff880036accf30 [44981.812277] ffff880036accf48 0000000000000202 0000000000000202 ffff88042ce7fa08 [44981.837303] Call Trace: [44981.845976] [] dump_stack+0x19/0x1b [44981.863518] [] dump_header+0x90/0x229 [44981.881624] [] ? ktime_get_ts64+0x4c/0xf0 [44981.900878] [] ? delayacct_end+0x8f/0xb0 [44981.919846] [] oom_kill_process+0x254/0x3d0 [44981.939673] [] ? oom_unkillable_task+0xcd/0x120 [44981.960651] [] ? find_lock_task_mm+0x56/0xc0 [44981.980759] [] out_of_memory+0x4b6/0x4f0 [44981.999710] [] __alloc_pages_slowpath+0x5d6/0x724 [44982.021242] [] __alloc_pages_nodemask+0x405/0x420 [44982.042772] [] alloc_pages_vma+0xb5/0x200 [44982.062013] [] read_swap_cache_async+0xed/0x160 [44982.082969] [] ? mem_cgroup_update_page_stat+0x16/0x50 [44982.105932] [] swapin_readahead+0xa8/0x110 [44982.125460] [] handle_mm_fault+0xadb/0xfa0 [44982.144987] [] __do_page_fault+0x154/0x450 [44982.164514] [] do_page_fault+0x35/0x90 [44982.182898] [] page_fault+0x28/0x30 [44982.200419] Mem-Info: [44982.208501] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:94 inactive_file:162 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9792 slab_unreclaimable:234971 mapped:1548 shmem:0 pagetables:1066 bounce:0 free:39349 free_pcp:295 free_cma:0 [44982.318773] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44982.457843] lowmem_reserve[]: 0 2580 15620 15620 [44982.473843] Node 0 DMA32 free:59372kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59788kB kernel_stack:1104kB pagetables:216kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2936 all_unreclaimable? yes [44982.623058] lowmem_reserve[]: 0 0 13040 13040 [44982.638319] Node 0 Normal free:36492kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:48kB inactive_file:148kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3796kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312132kB kernel_stack:10720kB pagetables:1524kB unstable:0kB bounce:0kB free_pcp:852kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1488 all_unreclaimable? yes [44982.793770] lowmem_reserve[]: 0 0 0 0 [44982.806772] Node 1 Normal free:45708kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20544kB slab_unreclaimable:567956kB kernel_stack:18864kB pagetables:2240kB unstable:0kB bounce:0kB free_pcp:328kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1363 all_unreclaimable? yes [44982.963139] lowmem_reserve[]: 0 0 0 0 [44982.976173] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44983.021311] Node 0 DMA32: 431*4kB (UEM) 712*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59372kB [44983.078035] Node 0 Normal: 125*4kB (UEM) 323*8kB (UEM) 580*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36492kB [44983.129078] Node 1 Normal: 1043*4kB (EM) 726*8kB (UEM) 375*16kB (UEM) 337*32kB (UEM) 268*64kB (UEM) 14*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45708kB [44983.180481] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44983.210241] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44983.239147] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44983.268898] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44983.297780] 1801 total pagecache pages [44983.311215] 38 pages in swap cache [44983.323496] Swap cache stats: add 70576, delete 70559, find 14915/20246 [44983.346377] Free swap = 16257200kB [44983.358937] Total swap = 16319484kB [44983.371518] 8369063 pages RAM [44983.382378] 0 pages HighMem/MovableOnly [44983.396097] 241261 pages reserved [44983.408094] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44983.434995] [ 1003] 0 1003 24572 207 51 93 0 systemd-journal [44983.465049] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44983.492804] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44983.522276] [ 1081] 0 1081 1588718 3982 133 0 -1000 multipathd [44983.550909] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44983.578364] [ 2588] 81 2588 9199 175 21 141 -900 dbus-daemon [44983.607221] [ 2621] 0 2621 64917 322 78 281 0 sssd [44983.634062] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44983.661734] [ 2637] 0 2637 5461 184 16 149 0 irqbalance [44983.690253] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44983.717610] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44983.745559] [ 2955] 0 2955 6051 221 16 83 0 systemd-logind [44983.775154] [ 3689] 996 3689 24211 171 21 130 0 munged [44983.802441] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44983.829704] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44983.856392] [ 3859] 0 3859 22907 166 44 269 0 master [44983.883614] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44983.910237] [ 3883] 0 3883 6464 163 19 52 0 atd [44983.936557] [ 3900] 0 3900 151290 201 68 379 0 automount [44983.964583] [ 3922] 0 3922 31559 185 19 161 0 crond [44983.991445] [ 3971] 0 3971 27511 156 12 32 0 agetty [44984.018983] [ 3975] 0 3975 27511 160 10 33 0 agetty [44984.046529] [147960] 89 147960 25474 209 46 256 0 pickup [44984.074544] [151569] 0 151569 40498 272 78 312 0 sshd [44984.101983] [151570] 74 151570 27583 151 56 276 0 sshd [44984.129363] [151571] 0 151571 79 0 4 12 0 systemd-cgroups [44984.159952] [151572] 0 151572 79 0 4 14 0 systemd-cgroups [44984.190427] [151574] 0 151574 610 26 4 13 0 systemd-cgroups [44984.220564] [151576] 0 151576 1 1 1 0 0 systemd-cgroups [44984.250673] Out of memory: Kill process 2621 (sssd) score 0 or sacrifice child [44984.275045] Killed process 2621 (sssd) total-vm:259668kB, anon-rss:0kB, file-rss:1288kB, shmem-rss:0kB [44984.428202] crond invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [44984.453721] crond cpuset=/ mems_allowed=0-1 [44984.468423] CPU: 0 PID: 3922 Comm: crond Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44984.504005] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44984.541876] ffff88083f6a8fd0 0000000030fd08d3 ffff88081571f920 ffffffff816a3e2d [44984.567241] ffff88081571f9b0 ffffffff8169f246 ffffffff810e93ac ffff88082ba398a0 [44984.592610] ffff88082ba398b8 0000000000000206 0000000000000206 ffff88081571f9a0 [44984.617994] Call Trace: [44984.626985] [] dump_stack+0x19/0x1b [44984.644868] [] dump_header+0x90/0x229 [44984.663299] [] ? ktime_get_ts64+0x4c/0xf0 [44984.682861] [] ? delayacct_end+0x8f/0xb0 [44984.702156] [] oom_kill_process+0x254/0x3d0 [44984.722311] [] ? oom_unkillable_task+0xcd/0x120 [44984.743618] [] ? find_lock_task_mm+0x56/0xc0 [44984.764062] [] out_of_memory+0x4b6/0x4f0 [44984.783360] [] __alloc_pages_slowpath+0x5d6/0x724 [44984.805221] [] __alloc_pages_nodemask+0x405/0x420 [44984.827063] [] alloc_pages_current+0x98/0x110 [44984.847766] [] __page_cache_alloc+0x97/0xb0 [44984.867902] [] filemap_fault+0x170/0x410 [44984.887184] [] ext4_filemap_fault+0x36/0x50 [ext4] [44984.909320] [] __do_fault+0x52/0xe0 [44984.927154] [] do_read_fault.isra.44+0x4b/0x130 [44984.948435] [] handle_mm_fault+0x691/0xfa0 [44984.968278] [] ? change_protection+0x65/0xa0 [44984.988693] [] __do_page_fault+0x154/0x450 [44985.008538] [] do_page_fault+0x35/0x90 [44985.027237] [] page_fault+0x28/0x30 [44985.045064] Mem-Info: [44985.053454] active_anon:0 inactive_anon:5 isolated_anon:0 active_file:90 inactive_file:155 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9792 slab_unreclaimable:234961 mapped:1548 shmem:0 pagetables:996 bounce:0 free:39378 free_pcp:0 free_cma:0 [44985.164778] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44985.305285] lowmem_reserve[]: 0 2580 15620 15620 [44985.321660] Node 0 DMA32 free:59396kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:12kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59788kB kernel_stack:1104kB pagetables:196kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:584 all_unreclaimable? yes [44985.472574] lowmem_reserve[]: 0 0 13040 13040 [44985.488236] Node 0 Normal free:36712kB min:37012kB low:46264kB high:55516kB active_anon:4kB inactive_anon:8kB active_file:76kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3796kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312132kB kernel_stack:10688kB pagetables:1340kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:461 all_unreclaimable? yes [44985.644367] lowmem_reserve[]: 0 0 0 0 [44985.657802] Node 1 Normal free:45516kB min:45728kB low:57160kB high:68592kB active_anon:4kB inactive_anon:0kB active_file:512kB inactive_file:300kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20512kB slab_unreclaimable:567876kB kernel_stack:18864kB pagetables:2136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1901 all_unreclaimable? yes [44985.815984] lowmem_reserve[]: 0 0 0 0 [44985.829545] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44985.875810] Node 0 DMA32: 435*4kB (UEM) 713*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59396kB [44985.933711] Node 0 Normal: 178*4kB (EM) 322*8kB (UEM) 589*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36840kB [44985.985671] Node 1 Normal: 1031*4kB (EM) 725*8kB (UEM) 367*16kB (EM) 335*32kB (UEM) 268*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45588kB [44986.038080] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44986.068469] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44986.097592] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44986.127344] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44986.156224] 1790 total pagecache pages [44986.169658] 6 pages in swap cache [44986.181654] Swap cache stats: add 70642, delete 70636, find 14936/20280 [44986.204536] Free swap = 16258248kB [44986.217116] Total swap = 16319484kB [44986.229694] 8369063 pages RAM [44986.240553] 0 pages HighMem/MovableOnly [44986.254277] 241261 pages reserved [44986.266276] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44986.293185] [ 1003] 0 1003 24572 206 51 94 0 systemd-journal [44986.323233] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44986.350987] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44986.380457] [ 1081] 0 1081 1588718 3982 133 0 -1000 multipathd [44986.409079] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44986.436531] [ 2588] 81 2588 9199 175 21 142 -900 dbus-daemon [44986.465388] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44986.493094] [ 2637] 0 2637 5461 184 16 150 0 irqbalance [44986.521622] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44986.548990] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44986.576910] [ 2955] 0 2955 6051 220 16 85 0 systemd-logind [44986.606533] [ 3689] 996 3689 24211 171 21 130 0 munged [44986.633840] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44986.661129] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44986.687830] [ 3859] 0 3859 22907 165 44 270 0 master [44986.715073] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44986.741727] [ 3883] 0 3883 6464 163 19 52 0 atd [44986.768067] [ 3900] 0 3900 151290 201 68 380 0 automount [44986.796098] [ 3922] 0 3922 31559 185 19 161 0 crond [44986.822993] [ 3971] 0 3971 27511 156 12 32 0 agetty [44986.850136] [ 3975] 0 3975 27511 160 10 33 0 agetty [44986.877301] [147960] 89 147960 25474 209 46 256 0 pickup [44986.904976] [151569] 0 151569 40498 272 78 313 0 sshd [44986.932056] [151570] 74 151570 27583 151 56 276 0 sshd [44986.959111] [151571] 0 151571 79 0 4 14 0 systemd-cgroups [44986.989294] [151572] 0 151572 79 0 4 14 0 systemd-cgroups [44987.019469] [151574] 0 151574 610 26 4 18 0 systemd-cgroups [44987.049619] [151576] 0 151576 1 1 1 0 0 systemd-cgroups [44987.079746] Out of memory: Kill process 151569 (sshd) score 0 or sacrifice child [44987.104711] Killed process 151570 (sshd) total-vm:110332kB, anon-rss:0kB, file-rss:604kB, shmem-rss:0kB [44987.160060] automount invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44987.186698] automount cpuset=/ mems_allowed=0-1 [44987.202360] CPU: 13 PID: 3903 Comm: automount Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44987.239154] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44987.276792] ffff88082b990fd0 00000000dd676e1e ffff8808173d7988 ffffffff816a3e2d [44987.301870] ffff8808173d7a18 ffffffff8169f246 ffffffff810e93ac ffff88082bd7ed70 [44987.326962] ffff88082bd7ed88 0000000000000206 0000000000000206 ffff8808173d7a08 [44987.352067] Call Trace: [44987.360821] [] dump_stack+0x19/0x1b [44987.378418] [] dump_header+0x90/0x229 [44987.396593] [] ? ktime_get_ts64+0x4c/0xf0 [44987.415908] [] ? delayacct_end+0x8f/0xb0 [44987.434926] [] oom_kill_process+0x254/0x3d0 [44987.454809] [] ? oom_unkillable_task+0xcd/0x120 [44987.475839] [] ? find_lock_task_mm+0x56/0xc0 [44987.496010] [] out_of_memory+0x4b6/0x4f0 [44987.515037] [] __alloc_pages_slowpath+0x5d6/0x724 [44987.536636] [] __alloc_pages_nodemask+0x405/0x420 [44987.558225] [] alloc_pages_vma+0xb5/0x200 [44987.577523] [] read_swap_cache_async+0xed/0x160 [44987.598531] [] swapin_readahead+0xa8/0x110 [44987.618115] [] handle_mm_fault+0xadb/0xfa0 [44987.637697] [] __do_page_fault+0x154/0x450 [44987.657269] [] do_page_fault+0x35/0x90 [44987.675692] [] page_fault+0x28/0x30 [44987.693264] Mem-Info: [44987.701386] active_anon:6 inactive_anon:8 isolated_anon:0 active_file:180 inactive_file:76 isolated_file:0 unevictable:6965 dirty:0 writeback:8 unstable:0 slab_reclaimable:9784 slab_unreclaimable:234921 mapped:1553 shmem:0 pagetables:870 bounce:0 free:38956 free_pcp:388 free_cma:0 [44987.811629] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44987.950870] lowmem_reserve[]: 0 2580 15620 15620 [44987.966919] Node 0 DMA32 free:59404kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:36kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59780kB kernel_stack:1104kB pagetables:196kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:399 all_unreclaimable? yes [44988.116325] lowmem_reserve[]: 0 0 13040 13040 [44988.131637] Node 0 Normal free:36872kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:396kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:4kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312124kB kernel_stack:10688kB pagetables:1336kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:597 all_unreclaimable? yes [44988.286345] lowmem_reserve[]: 0 0 0 0 [44988.299457] Node 1 Normal free:43972kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:344kB inactive_file:548kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1844kB shmem:0kB slab_reclaimable:20512kB slab_unreclaimable:567780kB kernel_stack:18896kB pagetables:1948kB unstable:0kB bounce:0kB free_pcp:1544kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2178 all_unreclaimable? yes [44988.456651] lowmem_reserve[]: 0 0 0 0 [44988.469794] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44988.515152] Node 0 DMA32: 435*4kB (UEM) 714*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59404kB [44988.572148] Node 0 Normal: 184*4kB (UEM) 323*8kB (UEM) 589*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36872kB [44988.623457] Node 1 Normal: 735*4kB (UEM) 725*8kB (UEM) 368*16kB (UEM) 321*32kB (UEM) 268*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 43972kB [44988.675121] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44988.705018] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44988.734051] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44988.763931] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44988.792942] 1812 total pagecache pages [44988.806506] 0 pages in swap cache [44988.818634] Swap cache stats: add 70690, delete 70690, find 14946/20299 [44988.841642] Free swap = 16258556kB [44988.854375] Total swap = 16319484kB [44988.867081] 8369063 pages RAM [44988.878077] 0 pages HighMem/MovableOnly [44988.891922] 241261 pages reserved [44988.904048] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44988.931077] [ 1003] 0 1003 24572 206 51 94 0 systemd-journal [44988.961251] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44988.989132] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44989.018732] [ 1081] 0 1081 1588718 3982 133 0 -1000 multipathd [44989.047493] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44989.075079] [ 2588] 81 2588 9199 175 21 143 -900 dbus-daemon [44989.104058] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44989.131879] [ 2637] 0 2637 5461 184 16 150 0 irqbalance [44989.160511] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44989.187990] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44989.216044] [ 2955] 0 2955 6051 220 16 85 0 systemd-logind [44989.245745] [ 3689] 996 3689 24211 171 21 133 0 munged [44989.273171] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44989.300567] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44989.327372] [ 3859] 0 3859 22907 164 44 270 0 master [44989.354721] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44989.381469] [ 3883] 0 3883 6464 163 19 52 0 atd [44989.407911] [ 3900] 0 3900 151290 201 68 380 0 automount [44989.436034] [ 3922] 0 3922 31559 185 19 163 0 crond [44989.463007] [ 3971] 0 3971 27511 156 12 32 0 agetty [44989.490236] [ 3975] 0 3975 27511 160 10 33 0 agetty [44989.517482] [147960] 89 147960 25474 209 46 256 0 pickup [44989.545239] [151569] 0 151569 40498 271 78 314 0 sshd [44989.572401] [151571] 0 151571 79 0 4 15 0 systemd-cgroups [44989.602693] [151572] 0 151572 79 0 4 15 0 systemd-cgroups [44989.632963] [151574] 0 151574 610 26 4 18 0 systemd-cgroups [44989.663219] [151576] 0 151576 76 0 3 2 0 systemd-cgroups [44989.693439] [151577] 0 151577 76 0 3 2 0 systemd-cgroups [44989.723640] [151578] 0 151578 76 0 3 2 0 systemd-cgroups [44989.753800] Out of memory: Kill process 151569 (sshd) score 0 or sacrifice child [44989.778783] Killed process 151569 (sshd) total-vm:161992kB, anon-rss:0kB, file-rss:1084kB, shmem-rss:0kB [44989.835236] crond invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [44989.860167] crond cpuset=/ mems_allowed=0-1 [44989.874483] CPU: 8 PID: 3922 Comm: crond Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44989.909685] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44989.947180] ffff88083f6a8fd0 0000000030fd08d3 ffff88081571f920 ffffffff816a3e2d [44989.972162] ffff88081571f9b0 ffffffff8169f246 ffffffff810e93ac ffff88082ba398a0 [44989.997183] ffff88082ba398b8 0000000000000206 0000000000000206 ffff88081571f9a0 [44990.022180] Call Trace: [44990.030808] [] dump_stack+0x19/0x1b [44990.048313] [] dump_header+0x90/0x229 [44990.066383] [] ? ktime_get_ts64+0x4c/0xf0 [44990.085573] [] ? delayacct_end+0x8f/0xb0 [44990.104497] [] oom_kill_process+0x254/0x3d0 [44990.124282] [] ? oom_unkillable_task+0xcd/0x120 [44990.145189] [] ? find_lock_task_mm+0x56/0xc0 [44990.165263] [] out_of_memory+0x4b6/0x4f0 [44990.184183] [] __alloc_pages_slowpath+0x5d6/0x724 [44990.205672] [] __alloc_pages_nodemask+0x405/0x420 [44990.227160] [] alloc_pages_current+0x98/0x110 [44990.247505] [] __page_cache_alloc+0x97/0xb0 [44990.267277] [] filemap_fault+0x170/0x410 [44990.286212] [] ext4_filemap_fault+0x36/0x50 [ext4] [44990.307985] [] __do_fault+0x52/0xe0 [44990.325472] [] do_read_fault.isra.44+0x4b/0x130 [44990.346389] [] handle_mm_fault+0x691/0xfa0 [44990.365877] [] __do_page_fault+0x154/0x450 [44990.385364] [] do_page_fault+0x35/0x90 [44990.403703] [] page_fault+0x28/0x30 [44990.421179] Mem-Info: [44990.429215] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:75 inactive_file:174 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9784 slab_unreclaimable:234875 mapped:1548 shmem:0 pagetables:792 bounce:0 free:39368 free_pcp:176 free_cma:0 [44990.538967] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44990.677984] lowmem_reserve[]: 0 2580 15620 15620 [44990.693931] Node 0 DMA32 free:59420kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:196kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:495 all_unreclaimable? yes [44990.842748] lowmem_reserve[]: 0 0 13040 13040 [44990.857978] Node 0 Normal free:36964kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:240kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312044kB kernel_stack:10688kB pagetables:1332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:638 all_unreclaimable? yes [44991.012024] lowmem_reserve[]: 0 0 0 0 [44991.024988] Node 1 Normal free:45512kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:376kB inactive_file:448kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1824kB shmem:0kB slab_reclaimable:20512kB slab_unreclaimable:567692kB kernel_stack:18896kB pagetables:1640kB unstable:0kB bounce:0kB free_pcp:836kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2148 all_unreclaimable? yes [44991.181249] lowmem_reserve[]: 0 0 0 0 [44991.194285] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44991.239487] Node 0 DMA32: 435*4kB (UEM) 716*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59420kB [44991.296216] Node 0 Normal: 187*4kB (UEM) 333*8kB (UEM) 589*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36964kB [44991.347248] Node 1 Normal: 889*4kB (UEM) 738*8kB (UEM) 369*16kB (UEM) 327*32kB (UEM) 268*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44900kB [44991.398652] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44991.428415] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44991.457321] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44991.487077] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44991.515959] 1803 total pagecache pages [44991.529393] 0 pages in swap cache [44991.541394] Swap cache stats: add 70743, delete 70743, find 14955/20315 [44991.564281] Free swap = 16259784kB [44991.576868] Total swap = 16319484kB [44991.589451] 8369063 pages RAM [44991.600289] 0 pages HighMem/MovableOnly [44991.614011] 241261 pages reserved [44991.626008] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44991.652918] [ 1003] 0 1003 24572 206 51 94 0 systemd-journal [44991.682965] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44991.710720] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44991.740191] [ 1081] 0 1081 1588718 3981 133 0 -1000 multipathd [44991.768813] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44991.796264] [ 2588] 81 2588 9199 174 21 143 -900 dbus-daemon [44991.825121] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44991.852814] [ 2637] 0 2637 5461 183 16 151 0 irqbalance [44991.881342] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44991.908724] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44991.936665] [ 2955] 0 2955 6051 219 16 85 0 systemd-logind [44991.966301] [ 3689] 996 3689 24211 171 21 133 0 munged [44991.993609] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44992.020897] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44992.047586] [ 3859] 0 3859 22907 163 44 270 0 master [44992.074837] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44992.101484] [ 3883] 0 3883 6464 163 19 52 0 atd [44992.127820] [ 3900] 0 3900 151290 201 68 381 0 automount [44992.155851] [ 3922] 0 3922 31559 184 19 163 0 crond [44992.182713] [ 3971] 0 3971 27511 156 12 32 0 agetty [44992.209858] [ 3975] 0 3975 27511 160 10 33 0 agetty [44992.237000] [147960] 89 147960 25474 209 46 256 0 pickup [44992.264679] [151571] 0 151571 79 0 4 15 0 systemd-cgroups [44992.294914] [151572] 0 151572 79 0 4 17 0 systemd-cgroups [44992.325123] [151574] 0 151574 610 26 4 19 0 systemd-cgroups [44992.355301] [151576] 0 151576 76 0 3 2 0 systemd-cgroups [44992.385470] [151577] 0 151577 79 0 3 5 0 systemd-cgroups [44992.415615] [151578] 0 151578 79 0 3 5 0 systemd-cgroups [44992.445730] Out of memory: Kill process 3900 (automount) score 0 or sacrifice child [44992.471546] Killed process 3900 (automount) total-vm:605160kB, anon-rss:0kB, file-rss:804kB, shmem-rss:0kB [44992.564174] systemd-journal invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [44992.592061] systemd-journal cpuset=/ mems_allowed=0-1 [44992.609341] CPU: 13 PID: 1003 Comm: systemd-journal Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44992.648419] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44992.686003] ffff880826db2f70 00000000cfeb2157 ffff88082a39b988 ffffffff816a3e2d [44992.711091] ffff88082a39ba18 ffffffff8169f246 ffffffff810e93ac ffff88082a7d5520 [44992.736194] ffff88082a7d5538 0000000000000202 0000000000000202 ffff88082a39ba08 [44992.761302] Call Trace: [44992.770037] [] dump_stack+0x19/0x1b [44992.787614] [] dump_header+0x90/0x229 [44992.805789] [] ? ktime_get_ts64+0x4c/0xf0 [44992.825077] [] ? delayacct_end+0x8f/0xb0 [44992.844108] [] oom_kill_process+0x254/0x3d0 [44992.863998] [] ? oom_unkillable_task+0xcd/0x120 [44992.885014] [] ? find_lock_task_mm+0x56/0xc0 [44992.905196] [] out_of_memory+0x4b6/0x4f0 [44992.924223] [] __alloc_pages_slowpath+0x5d6/0x724 [44992.945820] [] __alloc_pages_nodemask+0x405/0x420 [44992.967406] [] alloc_pages_vma+0xb5/0x200 [44992.986712] [] read_swap_cache_async+0xed/0x160 [44993.007731] [] swapin_readahead+0xa8/0x110 [44993.027319] [] handle_mm_fault+0xadb/0xfa0 [44993.046908] [] __do_page_fault+0x154/0x450 [44993.066488] [] do_page_fault+0x35/0x90 [44993.084922] [] page_fault+0x28/0x30 [44993.102475] Mem-Info: [44993.110610] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:93 inactive_file:153 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9784 slab_unreclaimable:234875 mapped:1547 shmem:0 pagetables:792 bounce:0 free:39340 free_pcp:1 free_cma:0 [44993.220330] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44993.359563] lowmem_reserve[]: 0 2580 15620 15620 [44993.375609] Node 0 DMA32 free:59428kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:188kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:306 all_unreclaimable? yes [44993.524744] lowmem_reserve[]: 0 0 13040 13040 [44993.540077] Node 0 Normal free:36816kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:284kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:312028kB kernel_stack:10624kB pagetables:1192kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:687 all_unreclaimable? yes [44993.694689] lowmem_reserve[]: 0 0 0 0 [44993.707821] Node 1 Normal free:45380kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:4kB active_file:324kB inactive_file:488kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:4kB mapped:1820kB shmem:0kB slab_reclaimable:20512kB slab_unreclaimable:567640kB kernel_stack:18864kB pagetables:1516kB unstable:0kB bounce:0kB free_pcp:124kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1877 all_unreclaimable? yes [44993.864620] lowmem_reserve[]: 0 0 0 0 [44993.877724] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44993.923036] Node 0 DMA32: 435*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59428kB [44993.979929] Node 0 Normal: 179*4kB (EM) 323*8kB (UEM) 593*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36916kB [44994.030880] Node 1 Normal: 1043*4kB (UEM) 724*8kB (EM) 367*16kB (EM) 327*32kB (UEM) 268*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45372kB [44994.082173] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44994.112023] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44994.140916] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44994.170662] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44994.199537] 1791 total pagecache pages [44994.212967] 0 pages in swap cache [44994.224965] Swap cache stats: add 70853, delete 70853, find 14982/20363 [44994.247846] Free swap = 16261200kB [44994.260429] Total swap = 16319484kB [44994.272984] 8369063 pages RAM [44994.283845] 0 pages HighMem/MovableOnly [44994.297562] 241261 pages reserved [44994.309562] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44994.336463] [ 1003] 0 1003 24572 206 51 94 0 systemd-journal [44994.366517] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44994.394270] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44994.423740] [ 1081] 0 1081 1588718 4047 133 0 -1000 multipathd [44994.452358] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44994.479804] [ 2588] 81 2588 9199 173 21 144 -900 dbus-daemon [44994.508666] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44994.536330] [ 2637] 0 2637 5461 181 16 153 0 irqbalance [44994.564834] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44994.592210] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44994.620155] [ 2955] 0 2955 6051 219 16 85 0 systemd-logind [44994.649820] [ 3689] 996 3689 24211 170 21 136 0 munged [44994.677125] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44994.704410] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44994.731110] [ 3859] 0 3859 22907 163 44 270 0 master [44994.758358] [ 3869] 89 3869 25491 211 49 279 0 qmgr [44994.784975] [ 3883] 0 3883 6464 163 19 52 0 atd [44994.811310] [ 3922] 0 3922 31559 183 19 164 0 crond [44994.838194] [ 3971] 0 3971 27511 156 12 32 0 agetty [44994.865338] [ 3975] 0 3975 27511 160 10 33 0 agetty [44994.892500] [147960] 89 147960 25474 209 46 256 0 pickup [44994.920198] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [44994.950452] [151572] 0 151572 79 0 4 17 0 systemd-cgroups [44994.980678] [151574] 0 151574 610 26 4 20 0 systemd-cgroups [44995.010872] [151576] 0 151576 76 0 3 2 0 systemd-cgroups [44995.041040] [151577] 0 151577 79 0 3 5 0 systemd-cgroups [44995.071201] [151578] 0 151578 79 0 3 5 0 systemd-cgroups [44995.101335] Out of memory: Kill process 3869 (qmgr) score 0 or sacrifice child [44995.125716] Killed process 3869 (qmgr) total-vm:101964kB, anon-rss:0kB, file-rss:844kB, shmem-rss:0kB [44995.174070] kworker/21:1 invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=0 [44995.200921] kworker/21:1 cpuset=/ mems_allowed=0-1 [44995.217416] CPU: 21 PID: 246 Comm: kworker/21:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44995.254736] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44995.292379] Workqueue: ib_cm cm_work_handler [ib_cm] [44995.309323] ffff88042c3ccf10 00000000eeeeb502 ffff88042bc0f578 ffffffff816a3e2d [44995.334359] ffff88042bc0f608 ffffffff8169f246 ffffffff810e93ac ffff88042c256130 [44995.359545] ffff88042c256148 0000000000000206 0000000000000206 ffff88042bc0f5f8 [44995.384609] Call Trace: [44995.393309] [] dump_stack+0x19/0x1b [44995.410853] [] dump_header+0x90/0x229 [44995.429001] [] ? ktime_get_ts64+0x4c/0xf0 [44995.448301] [] ? delayacct_end+0x8f/0xb0 [44995.467253] [] oom_kill_process+0x254/0x3d0 [44995.487072] [] ? oom_unkillable_task+0xcd/0x120 [44995.508133] [] ? find_lock_task_mm+0x56/0xc0 [44995.528222] [] out_of_memory+0x4b6/0x4f0 [44995.547203] [] __alloc_pages_slowpath+0x5d6/0x724 [44995.568766] [] __alloc_pages_nodemask+0x405/0x420 [44995.590370] [] dma_generic_alloc_coherent+0x8f/0x140 [44995.612794] [] x86_swiotlb_alloc_coherent+0x21/0x50 [44995.635023] [] mlx4_buf_alloc+0xd9/0x260 [mlx4_core] [44995.657393] [] mlx4_ib_alloc_cq_buf.isra.4+0x3e/0xe0 [mlx4_ib] [44995.682658] [] mlx4_ib_create_cq+0x21f/0x430 [mlx4_ib] [44995.705652] [] ? kiblnd_cq_completion+0x160/0x160 [ko2iblnd] [44995.730333] [] ? kiblnd_qp_event+0x150/0x150 [ko2iblnd] [44995.753632] [] ib_create_cq+0x29/0x60 [ib_core] [44995.774587] [] kiblnd_create_conn+0xb15/0x1960 [ko2iblnd] [44995.798501] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [44995.821271] [] kiblnd_passive_connect+0xa99/0x18c0 [ko2iblnd] [44995.846292] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [44995.869691] [] kiblnd_cm_callback+0x755/0x2380 [ko2iblnd] [44995.893559] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [44995.915952] [] cm_process_work+0x27/0x120 [ib_cm] [44995.937506] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [44995.959094] [] ? dequeue_task_fair+0x41e/0x660 [44995.979844] [] cm_work_handler+0x395/0x1306 [ib_cm] [44996.001978] [] process_one_work+0x17a/0x440 [44996.021879] [] worker_thread+0x126/0x3c0 [44996.040868] [] ? manage_workers.isra.24+0x2a0/0x2a0 [44996.063050] [] kthread+0xcf/0xe0 [44996.079804] [] ? insert_kthread_work+0x40/0x40 [44996.100639] [] ret_from_fork+0x58/0x90 [44996.119143] [] ? insert_kthread_work+0x40/0x40 [44996.139911] Mem-Info: [44996.148161] active_anon:0 inactive_anon:4 isolated_anon:0 active_file:163 inactive_file:163 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9784 slab_unreclaimable:234804 mapped:1567 shmem:0 pagetables:677 bounce:0 free:38963 free_pcp:0 free_cma:0 [44996.258281] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [44996.397302] lowmem_reserve[]: 0 2580 15620 15620 [44996.413319] Node 0 DMA32 free:59428kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:188kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1786 all_unreclaimable? yes [44996.562365] lowmem_reserve[]: 0 0 13040 13040 [44996.577650] Node 0 Normal free:36664kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:32kB inactive_file:132kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311852kB kernel_stack:10656kB pagetables:1004kB unstable:0kB bounce:0kB free_pcp:244kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:790 all_unreclaimable? yes [44996.732763] lowmem_reserve[]: 0 0 0 0 [44996.745779] Node 1 Normal free:45544kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1852kB shmem:0kB slab_reclaimable:20512kB slab_unreclaimable:567600kB kernel_stack:18864kB pagetables:1516kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1550 all_unreclaimable? yes [44996.902035] lowmem_reserve[]: 0 0 0 0 [44996.915240] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [44996.960583] Node 0 DMA32: 435*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59428kB [44997.017579] Node 0 Normal: 155*4kB (EM) 320*8kB (UEM) 575*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36540kB [44997.068558] Node 1 Normal: 1046*4kB (UEM) 732*8kB (UEM) 371*16kB (UEM) 328*32kB (UEM) 268*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45544kB [44997.120498] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44997.150369] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44997.179345] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44997.209136] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44997.238053] 1793 total pagecache pages [44997.251602] 0 pages in swap cache [44997.263703] Swap cache stats: add 70923, delete 70923, find 15003/20395 [44997.286668] Free swap = 16262204kB [44997.299332] Total swap = 16319484kB [44997.311990] 8369063 pages RAM [44997.322911] 0 pages HighMem/MovableOnly [44997.336757] 241261 pages reserved [44997.348764] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44997.375796] [ 1003] 0 1003 24572 206 51 95 0 systemd-journal [44997.405966] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44997.433809] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44997.463391] [ 1081] 0 1081 1588718 4006 133 0 -1000 multipathd [44997.492119] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44997.519658] [ 2588] 81 2588 9199 205 21 137 -900 dbus-daemon [44997.548558] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44997.576278] [ 2637] 0 2637 5461 181 16 153 0 irqbalance [44997.604913] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44997.632418] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44997.660390] [ 2955] 0 2955 6051 219 16 85 0 systemd-logind [44997.690096] [ 3689] 996 3689 24211 164 21 141 0 munged [44997.717439] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44997.744767] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44997.771534] [ 3859] 0 3859 22907 163 44 270 0 master [44997.798775] [ 3883] 0 3883 6464 163 19 52 0 atd [44997.825134] [ 3922] 0 3922 31559 183 19 165 0 crond [44997.852039] [ 3971] 0 3971 27511 156 12 32 0 agetty [44997.879206] [ 3975] 0 3975 27511 160 10 33 0 agetty [44997.906393] [147960] 89 147960 25474 209 46 257 0 pickup [44997.934117] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [44997.964393] [151572] 0 151572 79 0 4 20 0 systemd-cgroups [44997.994641] [151574] 0 151574 610 26 4 21 0 systemd-cgroups [44998.024865] [151576] 0 151576 79 0 4 9 0 systemd-cgroups [44998.055055] [151577] 0 151577 79 0 3 5 0 systemd-cgroups [44998.085215] [151578] 0 151578 79 0 4 6 0 systemd-cgroups [44998.115367] Out of memory: Kill process 147960 (pickup) score 0 or sacrifice child [44998.140937] Killed process 147960 (pickup) total-vm:101896kB, anon-rss:0kB, file-rss:836kB, shmem-rss:0kB [44998.214296] ll_ost_out01_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [44998.242642] ll_ost_out01_00 cpuset=/ mems_allowed=1 [44998.259771] CPU: 15 PID: 32074 Comm: ll_ost_out01_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [44998.299919] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [44998.337887] ffff88080801bf40 000000004282fc3a ffff8803c00ab998 ffffffff816a3e2d [44998.363303] ffff8803c00aba28 ffffffff8169f246 ffffffff810e93ac ffff8804183a1600 [44998.388712] ffff8804183a1618 0000000000000206 0000000000000206 ffff8803c00aba18 [44998.414127] Call Trace: [44998.423179] [] dump_stack+0x19/0x1b [44998.441130] [] dump_header+0x90/0x229 [44998.459620] [] ? ktime_get_ts64+0x4c/0xf0 [44998.479253] [] ? delayacct_end+0x8f/0xb0 [44998.498589] [] oom_kill_process+0x254/0x3d0 [44998.518779] [] ? oom_unkillable_task+0xcd/0x120 [44998.540117] [] ? find_lock_task_mm+0x56/0xc0 [44998.560601] [] out_of_memory+0x4b6/0x4f0 [44998.579940] [] __alloc_pages_slowpath+0x5d6/0x724 [44998.601860] [] __alloc_pages_nodemask+0x405/0x420 [44998.623449] [] __vmalloc_node_range+0x12f/0x270 [44998.644522] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44998.667829] [] vzalloc_node+0x47/0x50 [44998.686024] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44998.709360] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [44998.732127] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [44998.755494] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [44998.776833] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [44998.801875] [] kthread+0xcf/0xe0 [44998.818624] [] ? insert_kthread_work+0x40/0x40 [44998.839398] [] ret_from_fork+0x58/0x90 [44998.857868] [] ? insert_kthread_work+0x40/0x40 [44998.878632] Mem-Info: [44998.886800] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:153 inactive_file:95 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234798 mapped:1547 shmem:0 pagetables:631 bounce:0 free:39299 free_pcp:62 free_cma:0 [44998.997035] Node 1 Normal free:45408kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:448kB inactive_file:364kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567576kB kernel_stack:18864kB pagetables:1332kB unstable:0kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1368 all_unreclaimable? yes [44999.152770] lowmem_reserve[]: 0 0 0 0 [44999.165736] Node 1 Normal: 1013*4kB (EM) 724*8kB (UEM) 367*16kB (UEM) 326*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45284kB [44999.216792] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44999.246353] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44999.275031] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [44999.304529] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [44999.333171] 1793 total pagecache pages [44999.346328] 0 pages in swap cache [44999.358041] Swap cache stats: add 70938, delete 70938, find 15005/20400 [44999.380622] Free swap = 16263228kB [44999.392876] Total swap = 16319484kB [44999.405123] 8369063 pages RAM [44999.415645] 0 pages HighMem/MovableOnly [44999.429021] 241261 pages reserved [44999.440685] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [44999.467253] [ 1003] 0 1003 24572 206 51 95 0 systemd-journal [44999.496978] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [44999.524423] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [44999.553579] [ 1081] 0 1081 1588718 4006 133 0 -1000 multipathd [44999.581884] [ 2539] 0 2539 13863 114 26 134 -1000 auditd [44999.609035] [ 2588] 81 2588 9199 205 21 137 -900 dbus-daemon [44999.637616] [ 2630] 998 2630 29553 238 28 123 0 chronyd [44999.665047] [ 2637] 0 2637 5461 181 16 153 0 irqbalance [44999.693343] [ 2642] 0 2642 1770 27 8 37 0 mcelog [44999.720494] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [44999.748207] [ 2955] 0 2955 6051 219 16 85 0 systemd-logind [44999.777636] [ 3689] 996 3689 24211 164 21 141 0 munged [44999.804770] [ 3694] 0 3694 6779 148 18 62 0 xinetd [44999.831916] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [44999.858486] [ 3859] 0 3859 22907 163 44 270 0 master [44999.885609] [ 3883] 0 3883 6464 163 19 52 0 atd [44999.911863] [ 3922] 0 3922 31559 183 19 165 0 crond [44999.938690] [ 3971] 0 3971 27511 156 12 32 0 agetty [44999.965798] [ 3975] 0 3975 27511 160 10 33 0 agetty [44999.992947] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [45000.023200] [151572] 0 151572 79 0 4 20 0 systemd-cgroups [45000.053458] [151574] 0 151574 610 26 4 22 0 systemd-cgroups [45000.083701] [151576] 0 151576 79 0 4 9 0 systemd-cgroups [45000.113941] [151577] 0 151577 79 0 3 5 0 systemd-cgroups [45000.144168] [151578] 0 151578 79 0 4 6 0 systemd-cgroups [45000.174382] Out of memory: Kill process 3859 (master) score 0 or sacrifice child [45000.199462] Killed process 3859 (master) total-vm:91628kB, anon-rss:0kB, file-rss:652kB, shmem-rss:0kB [45000.253721] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45000.281863] systemd-cgroups cpuset=/ mems_allowed=0-1 [45000.299222] CPU: 17 PID: 151571 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45000.338997] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45000.376661] ffff880804038000 0000000075ff9a13 ffff88005e1db988 ffffffff816a3e2d [45000.401783] ffff88005e1dba18 ffffffff8169f246 ffffffff810e93ac ffff880409c05a60 [45000.426885] ffff88005e1dba68 0000000000000206 0000000000000001 0000000000003d13 [45000.451969] Call Trace: [45000.460706] [] dump_stack+0x19/0x1b [45000.478314] [] dump_header+0x90/0x229 [45000.496509] [] ? ktime_get_ts64+0x4c/0xf0 [45000.515826] [] oom_kill_process+0x254/0x3d0 [45000.535717] [] ? oom_unkillable_task+0xcd/0x120 [45000.556748] [] ? find_lock_task_mm+0x56/0xc0 [45000.576911] [] out_of_memory+0x4b6/0x4f0 [45000.595940] [] __alloc_pages_slowpath+0x5d6/0x724 [45000.617527] [] __alloc_pages_nodemask+0x405/0x420 [45000.639126] [] alloc_pages_vma+0xb5/0x200 [45000.658438] [] read_swap_cache_async+0xed/0x160 [45000.679475] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [45000.701920] [] swapin_readahead+0xa8/0x110 [45000.721514] [] handle_mm_fault+0xadb/0xfa0 [45000.741102] [] __do_page_fault+0x154/0x450 [45000.760681] [] do_page_fault+0x35/0x90 [45000.779112] [] page_fault+0x28/0x30 [45000.796682] Mem-Info: [45000.804809] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:89 inactive_file:159 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234795 mapped:1547 shmem:0 pagetables:587 bounce:0 free:39292 free_pcp:173 free_cma:0 [45000.915147] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45001.054432] lowmem_reserve[]: 0 2580 15620 15620 [45001.070477] Node 0 DMA32 free:59428kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:188kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3290 all_unreclaimable? yes [45001.219992] lowmem_reserve[]: 0 0 13040 13040 [45001.235328] Node 0 Normal free:36784kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:28kB inactive_file:136kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311848kB kernel_stack:10656kB pagetables:1004kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:509 all_unreclaimable? yes [45001.390443] lowmem_reserve[]: 0 0 0 0 [45001.403593] Node 1 Normal free:44760kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:376kB inactive_file:740kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1836kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567568kB kernel_stack:18864kB pagetables:1160kB unstable:0kB bounce:0kB free_pcp:688kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2230 all_unreclaimable? yes [45001.560459] lowmem_reserve[]: 0 0 0 0 [45001.573597] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45001.618975] Node 0 DMA32: 435*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59428kB [45001.675937] Node 0 Normal: 186*4kB (UEM) 320*8kB (EM) 583*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36792kB [45001.726945] Node 1 Normal: 1094*4kB (UEM) 728*8kB (UEM) 372*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45752kB [45001.778855] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45001.808731] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45001.837758] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45001.867646] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45001.896664] 1869 total pagecache pages [45001.910206] 0 pages in swap cache [45001.922313] Swap cache stats: add 70986, delete 70986, find 15018/20422 [45001.945322] Free swap = 16264276kB [45001.958020] Total swap = 16319484kB [45001.970708] 8369063 pages RAM [45001.981674] 0 pages HighMem/MovableOnly [45001.995503] 241261 pages reserved [45002.007600] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45002.034604] [ 1003] 0 1003 24572 206 51 95 0 systemd-journal [45002.064759] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [45002.092646] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45002.122236] [ 1081] 0 1081 1588718 4006 133 0 -1000 multipathd [45002.150989] [ 2539] 0 2539 13863 113 26 136 -1000 auditd [45002.178582] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45002.207558] [ 2630] 998 2630 29553 238 28 123 0 chronyd [45002.235367] [ 2637] 0 2637 5461 180 16 154 0 irqbalance [45002.264020] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45002.291503] [ 2944] 0 2944 30810 203 31 226 0 rpc.gssd [45002.307402] kworker/21:1: page allocation failure: order:8, mode:0x80d0 [45002.307404] CPU: 21 PID: 246 Comm: kworker/21:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45002.307405] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45002.307409] Workqueue: ib_cm cm_work_handler [ib_cm] [45002.307410] 00000000000080d0 00000000eeeeb502 ffff88042bc0f5a8 ffffffff816a3e2d [45002.307411] ffff88042bc0f638 ffffffff81188820 0000000000000000 00000000ffffffff [45002.307412] ffffffffffffff00 000080d000000000 ffff88042bc0f608 00000000eeeeb502 [45002.307413] Call Trace: [45002.307415] [] dump_stack+0x19/0x1b [45002.307418] [] warn_alloc_failed+0x110/0x180 [45002.307419] [] __alloc_pages_slowpath+0x6b6/0x724 [45002.307420] [] __alloc_pages_nodemask+0x405/0x420 [45002.307422] [] dma_generic_alloc_coherent+0x8f/0x140 [45002.307424] [] x86_swiotlb_alloc_coherent+0x21/0x50 [45002.307435] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [45002.307441] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [45002.307447] [] create_qp_common+0x536/0x1000 [mlx4_ib] [45002.307449] [] ? dma_pool_free+0xa7/0xd0 [45002.307453] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [45002.307457] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [45002.307462] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [45002.307468] [] ib_create_qp+0x7a/0x2f0 [ib_core] [45002.307471] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [45002.307476] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [45002.307483] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [45002.307487] [] kiblnd_passive_connect+0xa99/0x18c0 [ko2iblnd] [45002.307489] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [45002.307493] [] kiblnd_cm_callback+0x755/0x2380 [ko2iblnd] [45002.307495] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [45002.307497] [] cm_process_work+0x27/0x120 [ib_cm] [45002.307499] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [45002.307501] [] ? dequeue_task_fair+0x41e/0x660 [45002.307503] [] cm_work_handler+0x395/0x1306 [ib_cm] [45002.307505] [] process_one_work+0x17a/0x440 [45002.307506] [] worker_thread+0x126/0x3c0 [45002.307508] [] ? manage_workers.isra.24+0x2a0/0x2a0 [45002.307509] [] kthread+0xcf/0xe0 [45002.307511] [] ? insert_kthread_work+0x40/0x40 [45002.307513] [] ret_from_fork+0x58/0x90 [45002.307514] [] ? insert_kthread_work+0x40/0x40 [45002.307515] Mem-Info: [45002.307519] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:109 inactive_file:138 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234795 mapped:1547 shmem:0 pagetables:588 bounce:0 free:39387 free_pcp:0 free_cma:0 [45002.307522] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45002.307524] lowmem_reserve[]: 0 2580 15620 15620 [45002.307527] Node 0 DMA32 free:59428kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:188kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3702 all_unreclaimable? yes [45002.307528] lowmem_reserve[]: 0 0 13040 13040 [45002.307532] Node 0 Normal free:36792kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:108kB inactive_file:52kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311848kB kernel_stack:10656kB pagetables:1004kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:400 all_unreclaimable? yes [45002.307533] lowmem_reserve[]: 0 0 0 0 [45002.307536] Node 1 Normal free:45752kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567568kB kernel_stack:18864kB pagetables:1160kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [45002.307538] lowmem_reserve[]: 0 0 0 0 [45002.307545] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45002.307552] Node 0 DMA32: 435*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59428kB [45002.307556] Node 0 Normal: 186*4kB (UEM) 320*8kB (EM) 583*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36792kB [45002.307561] Node 1 Normal: 1064*4kB (UEM) 728*8kB (UEM) 372*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45632kB [45002.307562] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45002.307562] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45002.307563] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45002.307563] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45002.307564] 1792 total pagecache pages [45002.307564] 9 pages in swap cache [45002.307565] Swap cache stats: add 70995, delete 70986, find 15018/20423 [45002.307565] Free swap = 16264276kB [45002.307566] Total swap = 16319484kB [45002.307566] 8369063 pages RAM [45002.307567] 0 pages HighMem/MovableOnly [45002.307567] 241261 pages reserved [45002.309183] kworker/21:1: page allocation failure: order:8, mode:0x80d0 [45002.309184] CPU: 21 PID: 246 Comm: kworker/21:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45002.309185] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45002.309188] Workqueue: ib_cm cm_work_handler [ib_cm] [45002.309190] 00000000000080d0 00000000eeeeb502 ffff88042bc0f558 ffffffff816a3e2d [45002.309191] ffff88042bc0f5e8 ffffffff81188820 ffffffff8118b540 0000000000000000 [45002.309192] ffffffffffffff00 000080d000000000 ffff88042bc0f5b8 00000000eeeeb502 [45002.309192] Call Trace: [45002.309194] [] dump_stack+0x19/0x1b [45002.309196] [] warn_alloc_failed+0x110/0x180 [45002.309197] [] ? drain_pages+0xb0/0xb0 [45002.309198] [] __alloc_pages_slowpath+0x6b6/0x724 [45002.309199] [] __alloc_pages_nodemask+0x405/0x420 [45002.309201] [] alloc_pages_current+0x98/0x110 [45002.309203] [] __get_free_pages+0xe/0x40 [45002.309205] [] swiotlb_alloc_coherent+0x5e/0x150 [45002.309208] [] x86_swiotlb_alloc_coherent+0x41/0x50 [45002.309216] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [45002.309222] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [45002.309228] [] create_qp_common+0x536/0x1000 [mlx4_ib] [45002.309229] [] ? dma_pool_free+0xa7/0xd0 [45002.309233] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [45002.309238] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [45002.309243] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [45002.309249] [] ib_create_qp+0x7a/0x2f0 [ib_core] [45002.309251] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [45002.309254] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [45002.309260] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [45002.309264] [] kiblnd_passive_connect+0xa99/0x18c0 [ko2iblnd] [45002.309267] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [45002.309271] [] kiblnd_cm_callback+0x755/0x2380 [ko2iblnd] [45002.309275] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [45002.309277] [] cm_process_work+0x27/0x120 [ib_cm] [45002.309279] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [45002.309281] [] ? dequeue_task_fair+0x41e/0x660 [45002.309283] [] cm_work_handler+0x395/0x1306 [ib_cm] [45002.309284] [] process_one_work+0x17a/0x440 [45002.309286] [] worker_thread+0x126/0x3c0 [45002.309287] [] ? manage_workers.isra.24+0x2a0/0x2a0 [45002.309288] [] kthread+0xcf/0xe0 [45002.309289] [] ? insert_kthread_work+0x40/0x40 [45002.309291] [] ret_from_fork+0x58/0x90 [45002.309292] [] ? insert_kthread_work+0x40/0x40 [45002.309292] Mem-Info: [45002.309297] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:109 inactive_file:138 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234795 mapped:1547 shmem:0 pagetables:588 bounce:0 free:39387 free_pcp:0 free_cma:0 [45002.309302] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45002.309303] lowmem_reserve[]: 0 2580 15620 15620 [45002.309307] Node 0 DMA32 free:59428kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:188kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3702 all_unreclaimable? yes [45002.309308] lowmem_reserve[]: 0 0 13040 13040 [45002.309311] Node 0 Normal free:36792kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:108kB inactive_file:52kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311848kB kernel_stack:10656kB pagetables:1004kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:400 all_unreclaimable? yes [45002.309312] lowmem_reserve[]: 0 0 0 0 [45002.309315] Node 1 Normal free:45752kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567568kB kernel_stack:18864kB pagetables:1160kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:294 all_unreclaimable? no [45002.309316] lowmem_reserve[]: 0 0 0 0 [45002.309321] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45002.309328] Node 0 DMA32: 435*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59428kB [45002.309334] Node 0 Normal: 186*4kB (UEM) 320*8kB (EM) 583*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36792kB [45002.309338] Node 1 Normal: 1064*4kB (UEM) 728*8kB (UEM) 372*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45632kB [45002.309339] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45002.309339] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45002.309340] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45002.309341] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45002.309341] 1792 total pagecache pages [45002.309341] 9 pages in swap cache [45002.309342] Swap cache stats: add 70995, delete 70986, find 15018/20423 [45002.309342] Free swap = 16264276kB [45002.309343] Total swap = 16319484kB [45002.309343] 8369063 pages RAM [45002.309343] 0 pages HighMem/MovableOnly [45002.309344] 241261 pages reserved [45006.593742] [ 2955] 0 2955 6051 219 16 86 0 systemd-logind [45006.623587] [ 3689] 996 3689 24211 164 21 141 0 munged [45006.651131] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45006.678698] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45006.705688] [ 3883] 0 3883 6464 163 19 52 0 atd [45006.732386] [ 3922] 0 3922 31559 182 19 167 0 crond [45006.759636] [ 3971] 0 3971 27511 156 12 32 0 agetty [45006.787130] [ 3975] 0 3975 27511 160 10 33 0 agetty [45006.814652] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [45006.845262] [151572] 0 151572 79 0 4 20 0 systemd-cgroups [45006.875855] [151574] 0 151574 610 26 4 22 0 systemd-cgroups [45006.906419] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45006.936931] [151577] 0 151577 79 0 4 5 0 systemd-cgroups [45006.967417] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45006.997862] Out of memory: Kill process 2944 (rpc.gssd) score 0 or sacrifice child [45007.023739] Killed process 2944 (rpc.gssd) total-vm:123240kB, anon-rss:0kB, file-rss:812kB, shmem-rss:0kB [45007.081082] crond invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45007.106623] crond cpuset=/ mems_allowed=0-1 [45007.121338] CPU: 30 PID: 3922 Comm: crond Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45007.157118] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45007.194890] ffff88083f6a8fd0 0000000030fd08d3 ffff88081571f988 ffffffff816a3e2d [45007.220138] ffff88081571fa18 ffffffff8169f246 ffffffff810e93ac ffff88082ba398a0 [45007.245366] ffff88082ba398b8 0000000000000202 0000000000000202 ffff88081571fa08 [45007.270576] Call Trace: [45007.279410] [] dump_stack+0x19/0x1b [45007.297094] [] dump_header+0x90/0x229 [45007.315327] [] ? ktime_get_ts64+0x4c/0xf0 [45007.334702] [] ? delayacct_end+0x8f/0xb0 [45007.353773] [] oom_kill_process+0x254/0x3d0 [45007.373677] [] ? oom_unkillable_task+0xcd/0x120 [45007.394707] [] ? find_lock_task_mm+0x56/0xc0 [45007.414858] [] out_of_memory+0x4b6/0x4f0 [45007.433814] [] __alloc_pages_slowpath+0x5d6/0x724 [45007.455355] [] __alloc_pages_nodemask+0x405/0x420 [45007.476871] [] alloc_pages_vma+0xb5/0x200 [45007.496078] [] read_swap_cache_async+0xed/0x160 [45007.516984] [] swapin_readahead+0xa8/0x110 [45007.536468] [] handle_mm_fault+0xadb/0xfa0 [45007.555959] [] __do_page_fault+0x154/0x450 [45007.575451] [] do_page_fault+0x35/0x90 [45007.593789] [] page_fault+0x28/0x30 [45007.611268] Mem-Info: [45007.619301] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:99 inactive_file:147 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234787 mapped:1547 shmem:0 pagetables:557 bounce:0 free:39350 free_pcp:1 free_cma:0 [45007.728445] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45007.867312] lowmem_reserve[]: 0 2580 15620 15620 [45007.883230] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:294 all_unreclaimable? yes [45008.031979] lowmem_reserve[]: 0 0 13040 13040 [45008.047189] Node 0 Normal free:36836kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:48kB inactive_file:108kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311840kB kernel_stack:10656kB pagetables:972kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:468 all_unreclaimable? yes [45008.201355] lowmem_reserve[]: 0 0 0 0 [45008.214350] Node 1 Normal free:45556kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:340kB inactive_file:472kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567544kB kernel_stack:18864kB pagetables:1072kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1601 all_unreclaimable? yes [45008.370104] lowmem_reserve[]: 0 0 0 0 [45008.383137] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45008.428264] Node 0 DMA32: 436*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45008.484985] Node 0 Normal: 184*4kB (EM) 321*8kB (EM) 578*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36712kB [45008.535450] Node 1 Normal: 1071*4kB (UEM) 725*8kB (UEM) 367*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45556kB [45008.587129] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45008.616882] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45008.645770] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45008.675513] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45008.704387] 1791 total pagecache pages [45008.717814] 0 pages in swap cache [45008.729805] Swap cache stats: add 71006, delete 71006, find 15022/20430 [45008.752679] Free swap = 16265176kB [45008.765257] Total swap = 16319484kB [45008.777834] 8369063 pages RAM [45008.788690] 0 pages HighMem/MovableOnly [45008.802403] 241261 pages reserved [45008.814397] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45008.841295] [ 1003] 0 1003 24572 205 51 95 0 systemd-journal [45008.871342] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [45008.899096] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45008.928564] [ 1081] 0 1081 1588718 4006 133 0 -1000 multipathd [45008.957182] [ 2539] 0 2539 13863 113 26 136 -1000 auditd [45008.984640] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45009.013494] [ 2630] 998 2630 29553 238 28 123 0 chronyd [45009.041182] [ 2637] 0 2637 5461 180 16 154 0 irqbalance [45009.069710] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45009.097077] [ 2955] 0 2955 6051 219 16 86 0 systemd-logind [45009.126731] [ 3689] 996 3689 24211 164 21 141 0 munged [45009.154058] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45009.181338] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45009.208050] [ 3883] 0 3883 6464 163 19 52 0 atd [45009.234450] [ 3922] 0 3922 31559 182 19 167 0 crond [45009.261417] [ 3971] 0 3971 27511 156 12 32 0 agetty [45009.288634] [ 3975] 0 3975 27511 160 10 33 0 agetty [45009.315853] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [45009.346178] [151572] 0 151572 79 0 4 20 0 systemd-cgroups [45009.376473] [151574] 0 151574 610 26 4 22 0 systemd-cgroups [45009.406758] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45009.437019] [151577] 0 151577 79 0 4 5 0 systemd-cgroups [45009.467251] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45009.497456] Out of memory: Kill process 2630 (chronyd) score 0 or sacrifice child [45009.522784] Killed process 2630 (chronyd) total-vm:118212kB, anon-rss:0kB, file-rss:952kB, shmem-rss:0kB [45009.576631] kworker/u481:18 invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 [45009.605011] kworker/u481:18 cpuset=/ mems_allowed=0-1 [45009.622451] CPU: 5 PID: 573 Comm: kworker/u481:18 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45009.660387] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45009.698098] Workqueue: events_unbound __call_usermodehelper [45009.717151] ffff88041876af70 000000001ba44681 ffff8804187b39a0 ffffffff816a3e2d [45009.742307] ffff8804187b3a30 ffffffff8169f246 ffffffff810e93ac ffff88042ba90520 [45009.767451] ffff88042ba90538 0000000000000206 0000000000000206 ffff8804187b3a20 [45009.792576] Call Trace: [45009.801252] [] dump_stack+0x19/0x1b [45009.818850] [] dump_header+0x90/0x229 [45009.837041] [] ? ktime_get_ts64+0x4c/0xf0 [45009.856375] [] ? delayacct_end+0x8f/0xb0 [45009.875346] [] oom_kill_process+0x254/0x3d0 [45009.895167] [] ? oom_unkillable_task+0xcd/0x120 [45009.916183] [] ? find_lock_task_mm+0x56/0xc0 [45009.936292] [] out_of_memory+0x4b6/0x4f0 [45009.955264] [] __alloc_pages_slowpath+0x5d6/0x724 [45009.976824] [] __alloc_pages_nodemask+0x405/0x420 [45009.998360] [] ? ____call_usermodehelper+0x130/0x130 [45010.020766] [] copy_process+0x1dd/0x1970 [45010.039741] [] ? account_entity_dequeue+0xae/0xd0 [45010.061302] [] do_fork+0x91/0x320 [45010.078241] [] ? __switch_to+0xd7/0x510 [45010.096902] [] kernel_thread+0x26/0x30 [45010.115271] [] __call_usermodehelper+0x2e/0x90 [45010.135992] [] process_one_work+0x17a/0x440 [45010.155837] [] worker_thread+0x126/0x3c0 [45010.174844] [] ? manage_workers.isra.24+0x2a0/0x2a0 [45010.196950] [] kthread+0xcf/0xe0 [45010.213612] [] ? insert_kthread_work+0x40/0x40 [45010.234285] [] ret_from_fork+0x58/0x90 [45010.252664] [] ? insert_kthread_work+0x40/0x40 [45010.273422] Mem-Info: [45010.281599] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:94 inactive_file:152 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234787 mapped:1547 shmem:0 pagetables:557 bounce:0 free:39319 free_pcp:2 free_cma:0 [45010.391143] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45010.530452] lowmem_reserve[]: 0 2580 15620 15620 [45010.546554] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1114 all_unreclaimable? yes [45010.695924] lowmem_reserve[]: 0 0 13040 13040 [45010.711145] Node 0 Normal free:36896kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:60kB inactive_file:96kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311836kB kernel_stack:10656kB pagetables:904kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:285 all_unreclaimable? yes [45010.864976] lowmem_reserve[]: 0 0 0 0 [45010.878030] Node 1 Normal free:45660kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567508kB kernel_stack:18864kB pagetables:1028kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1304 all_unreclaimable? yes [45011.034052] lowmem_reserve[]: 0 0 0 0 [45011.047148] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45011.092555] Node 0 DMA32: 436*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45011.149470] Node 0 Normal: 198*4kB (UEM) 327*8kB (UEM) 583*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36896kB [45011.200759] Node 1 Normal: 1087*4kB (UEM) 730*8kB (UEM) 367*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45660kB [45011.252707] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45011.282518] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45011.311516] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45011.341363] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45011.370360] 1791 total pagecache pages [45011.383907] 0 pages in swap cache [45011.395967] Swap cache stats: add 71008, delete 71008, find 15026/20435 [45011.418937] Free swap = 16265668kB [45011.431616] Total swap = 16319484kB [45011.444276] 8369063 pages RAM [45011.455227] 0 pages HighMem/MovableOnly [45011.469007] 241261 pages reserved [45011.481094] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45011.508053] [ 1003] 0 1003 24572 205 51 95 0 systemd-journal [45011.538165] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [45011.566039] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45011.595605] [ 1081] 0 1081 1588718 4006 133 0 -1000 multipathd [45011.624245] [ 2539] 0 2539 13863 113 26 136 -1000 auditd [45011.651728] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45011.680676] [ 2637] 0 2637 5461 180 16 154 0 irqbalance [45011.709316] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45011.736768] [ 2955] 0 2955 6051 219 16 86 0 systemd-logind [45011.766530] [ 3689] 996 3689 24211 164 21 141 0 munged [45011.793977] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45011.821381] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45011.848206] [ 3883] 0 3883 6464 163 19 52 0 atd [45011.874637] [ 3922] 0 3922 31559 182 19 167 0 crond [45011.901635] [ 3971] 0 3971 27511 156 12 32 0 agetty [45011.928895] [ 3975] 0 3975 27511 160 10 33 0 agetty [45011.956220] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [45011.986583] [151572] 0 151572 79 0 4 20 0 systemd-cgroups [45012.016973] [151574] 0 151574 610 26 4 22 0 systemd-cgroups [45012.047334] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45012.077719] [151577] 0 151577 79 0 4 5 0 systemd-cgroups [45012.108062] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45012.138307] Out of memory: Kill process 3922 (crond) score 0 or sacrifice child [45012.163152] Killed process 3922 (crond) total-vm:126236kB, anon-rss:0kB, file-rss:728kB, shmem-rss:0kB [45012.213381] systemd-journal invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45012.242093] systemd-journal cpuset=/ mems_allowed=0-1 [45012.259934] CPU: 13 PID: 1003 Comm: systemd-journal Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45012.300037] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45012.337774] ffff880826db2f70 00000000cfeb2157 ffff88082a39b988 ffffffff816a3e2d [45012.362951] ffff88082a39ba18 ffffffff8169f246 ffffffff810e93ac ffff88082a7d5520 [45012.388129] ffff88082a7d5538 0000000000000206 0000000000000206 ffff88082a39ba08 [45012.413269] Call Trace: [45012.422032] [] dump_stack+0x19/0x1b [45012.439640] [] dump_header+0x90/0x229 [45012.457804] [] ? ktime_get_ts64+0x4c/0xf0 [45012.477119] [] ? delayacct_end+0x8f/0xb0 [45012.496159] [] oom_kill_process+0x254/0x3d0 [45012.516058] [] ? oom_unkillable_task+0xcd/0x120 [45012.537089] [] ? find_lock_task_mm+0x56/0xc0 [45012.557272] [] out_of_memory+0x4b6/0x4f0 [45012.576297] [] __alloc_pages_slowpath+0x5d6/0x724 [45012.597899] [] __alloc_pages_nodemask+0x405/0x420 [45012.619499] [] alloc_pages_vma+0xb5/0x200 [45012.638808] [] read_swap_cache_async+0xed/0x160 [45012.659854] [] ? mem_cgroup_update_page_stat+0x16/0x50 [45012.682893] [] swapin_readahead+0xa8/0x110 [45012.702497] [] handle_mm_fault+0xadb/0xfa0 [45012.722056] [] __do_page_fault+0x154/0x450 [45012.741644] [] do_page_fault+0x35/0x90 [45012.760085] [] page_fault+0x28/0x30 [45012.777656] Mem-Info: [45012.785784] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:129 inactive_file:117 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234773 mapped:1547 shmem:0 pagetables:510 bounce:0 free:39306 free_pcp:0 free_cma:0 [45012.895829] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45013.035129] lowmem_reserve[]: 0 2580 15620 15620 [45013.051205] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1874 all_unreclaimable? yes [45013.200730] lowmem_reserve[]: 0 0 13040 13040 [45013.216085] Node 0 Normal free:36904kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:60kB inactive_file:96kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311828kB kernel_stack:10656kB pagetables:904kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:234 all_unreclaimable? yes [45013.371116] lowmem_reserve[]: 0 0 0 0 [45013.384234] Node 1 Normal free:45432kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:448kB inactive_file:364kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1824kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567468kB kernel_stack:18864kB pagetables:952kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1331 all_unreclaimable? yes [45013.540259] lowmem_reserve[]: 0 0 0 0 [45013.553398] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45013.598796] Node 0 DMA32: 436*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45013.655778] Node 0 Normal: 185*4kB (UEM) 321*8kB (EM) 582*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36780kB [45013.706802] Node 1 Normal: 1044*4kB (UEM) 723*8kB (EM) 367*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45432kB [45013.758463] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45013.788347] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45013.817374] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45013.847252] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45013.876272] 1792 total pagecache pages [45013.889824] 0 pages in swap cache [45013.901943] Swap cache stats: add 71018, delete 71018, find 15027/20438 [45013.924949] Free swap = 16266296kB [45013.937651] Total swap = 16319484kB [45013.950353] 8369063 pages RAM [45013.961337] 0 pages HighMem/MovableOnly [45013.975172] 241261 pages reserved [45013.987301] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45014.014337] [ 1003] 0 1003 24572 205 51 95 0 systemd-journal [45014.044529] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [45014.072429] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45014.102026] [ 1081] 0 1081 1588718 4004 133 0 -1000 multipathd [45014.130777] [ 2539] 0 2539 13863 113 26 136 -1000 auditd [45014.158364] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45014.187338] [ 2637] 0 2637 5461 180 16 154 0 irqbalance [45014.216013] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45014.243501] [ 2955] 0 2955 6051 219 16 86 0 systemd-logind [45014.273285] [ 3689] 996 3689 24211 164 21 141 0 munged [45014.300752] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45014.328196] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45014.355047] [ 3883] 0 3883 6464 163 19 52 0 atd [45014.381587] [ 3971] 0 3971 27511 156 12 32 0 agetty [45014.408945] [ 3975] 0 3975 27511 160 10 33 0 agetty [45014.436315] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [45014.466781] [151572] 0 151572 79 0 4 20 0 systemd-cgroups [45014.497219] [151574] 0 151574 610 26 4 22 0 systemd-cgroups [45014.527642] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45014.558020] [151577] 0 151577 79 0 4 5 0 systemd-cgroups [45014.588382] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45014.618731] Out of memory: Kill process 2637 (irqbalance) score 0 or sacrifice child [45014.645078] Killed process 2637 (irqbalance) total-vm:21844kB, anon-rss:0kB, file-rss:720kB, shmem-rss:0kB [45014.681138] auditd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=-1000 [45014.707760] auditd cpuset=/ mems_allowed=0-1 [45014.722634] CPU: 26 PID: 2539 Comm: auditd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45014.758631] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45014.796331] ffff880819babf40 000000006189215e ffff88081d8db988 ffffffff816a3e2d [45014.821641] ffff88081d8dba18 ffffffff8169f246 ffffffff810e93ac ffff88082bc4a520 [45014.846857] ffff88082bc4a538 0000000000000202 0000000000000202 ffff88081d8dba08 [45014.872082] Call Trace: [45014.880871] [] dump_stack+0x19/0x1b [45014.898473] [] dump_header+0x90/0x229 [45014.916676] [] ? ktime_get_ts64+0x4c/0xf0 [45014.935918] [] ? delayacct_end+0x8f/0xb0 [45014.954910] [] oom_kill_process+0x254/0x3d0 [45014.974752] [] ? oom_unkillable_task+0xcd/0x120 [45014.995774] [] ? find_lock_task_mm+0x56/0xc0 [45015.015958] [] out_of_memory+0x4b6/0x4f0 [45015.035007] [] __alloc_pages_slowpath+0x5d6/0x724 [45015.056576] [] __alloc_pages_nodemask+0x405/0x420 [45015.078097] [] alloc_pages_vma+0xb5/0x200 [45015.097345] [] read_swap_cache_async+0xed/0x160 [45015.118287] [] ? mem_cgroup_update_page_stat+0x16/0x50 [45015.141236] [] swapin_readahead+0xa8/0x110 [45015.160756] [] handle_mm_fault+0xadb/0xfa0 [45015.180274] [] __do_page_fault+0x154/0x450 [45015.199781] [] do_page_fault+0x35/0x90 [45015.218134] [] page_fault+0x28/0x30 [45015.235629] Mem-Info: [45015.243677] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:106 inactive_file:140 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234765 mapped:1548 shmem:0 pagetables:510 bounce:0 free:39335 free_pcp:115 free_cma:0 [45015.353749] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45015.492650] lowmem_reserve[]: 0 2580 15620 15620 [45015.508604] Node 0 DMA32 free:59432kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:184kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2686 all_unreclaimable? yes [45015.657681] lowmem_reserve[]: 0 0 13040 13040 [45015.672898] Node 0 Normal free:36900kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:96kB inactive_file:60kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311828kB kernel_stack:10656kB pagetables:840kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:313 all_unreclaimable? yes [45015.826788] lowmem_reserve[]: 0 0 0 0 [45015.839782] Node 1 Normal free:45060kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1924kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567468kB kernel_stack:18864kB pagetables:952kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1318 all_unreclaimable? yes [45015.995252] lowmem_reserve[]: 0 0 0 0 [45016.008284] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45016.053417] Node 0 DMA32: 436*4kB (UEM) 717*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45016.110123] Node 0 Normal: 215*4kB (UEM) 330*8kB (UEM) 583*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36988kB [45016.161153] Node 1 Normal: 1045*4kB (UEM) 723*8kB (EM) 367*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45436kB [45016.212561] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45016.242338] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45016.271234] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45016.300982] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45016.329863] 1792 total pagecache pages [45016.343292] 0 pages in swap cache [45016.355286] Swap cache stats: add 71034, delete 71034, find 15028/20440 [45016.378164] Free swap = 16266880kB [45016.390746] Total swap = 16319484kB [45016.403327] 8369063 pages RAM [45016.414184] 0 pages HighMem/MovableOnly [45016.427904] 241261 pages reserved [45016.439897] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45016.466797] [ 1003] 0 1003 24572 205 51 95 0 systemd-journal [45016.496843] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [45016.524594] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45016.554062] [ 1081] 0 1081 1588718 4023 133 0 -1000 multipathd [45016.582678] [ 2539] 0 2539 13863 113 26 136 -1000 auditd [45016.610122] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45016.638987] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45016.666398] [ 2955] 0 2955 6051 218 16 86 0 systemd-logind [45016.696070] [ 3689] 996 3689 24211 164 21 141 0 munged [45016.723439] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45016.750789] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45016.777542] [ 3883] 0 3883 6464 163 19 52 0 atd [45016.803987] [ 3971] 0 3971 27511 156 12 32 0 agetty [45016.831268] [ 3975] 0 3975 27511 160 10 33 0 agetty [45016.858559] [151571] 0 151571 79 0 4 18 0 systemd-cgroups [45016.888949] [151572] 0 151572 79 0 4 20 0 systemd-cgroups [45016.919310] [151574] 0 151574 610 26 4 22 0 systemd-cgroups [45016.949640] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45016.979944] [151577] 0 151577 79 0 4 5 0 systemd-cgroups [45017.010221] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45017.040488] Out of memory: Kill process 1003 (systemd-journal) score 0 or sacrifice child [45017.068175] Killed process 1003 (systemd-journal) total-vm:98288kB, anon-rss:0kB, file-rss:820kB, shmem-rss:0kB [45017.230942] kworker/u481:18 invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0 [45017.259498] kworker/u481:18 cpuset=/ mems_allowed=0-1 [45017.276960] CPU: 23 PID: 573 Comm: kworker/u481:18 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45017.316165] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45017.353952] Workqueue: events_unbound __call_usermodehelper [45017.373164] ffff88041876af70 000000001ba44681 ffff8804187b39a0 ffffffff816a3e2d [45017.398380] ffff8804187b3a30 ffffffff8169f246 ffffffff810e93ac ffff88042ba90520 [45017.423579] ffff88042ba90538 0000000000000206 0000000000000206 ffff8804187b3a20 [45017.448771] Call Trace: [45017.457564] [] dump_stack+0x19/0x1b [45017.475211] [] dump_header+0x90/0x229 [45017.493400] [] ? ktime_get_ts64+0x4c/0xf0 [45017.512732] [] ? delayacct_end+0x8f/0xb0 [45017.531778] [] oom_kill_process+0x254/0x3d0 [45017.551688] [] ? oom_unkillable_task+0xcd/0x120 [45017.572764] [] ? find_lock_task_mm+0x56/0xc0 [45017.592958] [] out_of_memory+0x4b6/0x4f0 [45017.612006] [] __alloc_pages_slowpath+0x5d6/0x724 [45017.633629] [] __alloc_pages_nodemask+0x405/0x420 [45017.655242] [] ? ____call_usermodehelper+0x130/0x130 [45017.677721] [] copy_process+0x1dd/0x1970 [45017.696766] [] ? account_entity_dequeue+0xae/0xd0 [45017.718395] [] do_fork+0x91/0x320 [45017.735448] [] ? __switch_to+0xd7/0x510 [45017.754205] [] kernel_thread+0x26/0x30 [45017.772663] [] __call_usermodehelper+0x2e/0x90 [45017.793407] [] process_one_work+0x17a/0x440 [45017.813294] [] worker_thread+0x126/0x3c0 [45017.832327] [] ? manage_workers.isra.24+0x2a0/0x2a0 [45017.854508] [] kthread+0xcf/0xe0 [45017.871264] [] ? insert_kthread_work+0x40/0x40 [45017.892027] [] ret_from_fork+0x58/0x90 [45017.910497] [] ? insert_kthread_work+0x40/0x40 [45017.931252] Mem-Info: [45017.939411] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:126 inactive_file:120 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9782 slab_unreclaimable:234756 mapped:1550 shmem:0 pagetables:443 bounce:0 free:39334 free_pcp:276 free_cma:0 [45018.050224] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45018.189764] lowmem_reserve[]: 0 2580 15620 15620 [45018.205885] Node 0 DMA32 free:59456kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:176kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:420 all_unreclaimable? yes [45018.355143] lowmem_reserve[]: 0 0 13040 13040 [45018.370430] Node 0 Normal free:35364kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:104kB inactive_file:236kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3828kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311820kB kernel_stack:10672kB pagetables:740kB unstable:0kB bounce:0kB free_pcp:1104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1050 all_unreclaimable? yes [45018.526160] lowmem_reserve[]: 0 0 0 0 [45018.539211] Node 1 Normal free:45560kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:404kB inactive_file:408kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1832kB shmem:0kB slab_reclaimable:20504kB slab_unreclaimable:567440kB kernel_stack:18864kB pagetables:856kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:12870 all_unreclaimable? yes [45018.695445] lowmem_reserve[]: 0 0 0 0 [45018.708631] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45018.754052] Node 0 DMA32: 440*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59456kB [45018.811031] Node 0 Normal: 45*4kB (EM) 252*8kB (UEM) 563*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35364kB [45018.861762] Node 1 Normal: 1068*4kB (UEM) 727*8kB (UEM) 367*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45560kB [45018.913706] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45018.943611] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45018.972648] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45019.002536] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45019.031563] 1842 total pagecache pages [45019.045132] 0 pages in swap cache [45019.057267] Swap cache stats: add 71091, delete 71091, find 15050/20470 [45019.080288] Free swap = 16267192kB [45019.093011] Total swap = 16319484kB [45019.105733] 8369063 pages RAM [45019.116722] 0 pages HighMem/MovableOnly [45019.130573] 241261 pages reserved [45019.142699] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45019.169730] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [45019.197609] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45019.227207] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [45019.255964] [ 2539] 0 2539 13863 113 26 137 -1000 auditd [45019.283527] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45019.312530] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45019.340081] [ 2955] 0 2955 6051 218 16 88 0 systemd-logind [45019.369893] [ 3689] 996 3689 24211 164 21 141 0 munged [45019.397392] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45019.424882] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45019.451777] [ 3883] 0 3883 6464 163 19 52 0 atd [45019.467076] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1513255129/real 0] req@ffff8803f57b4b00 x1586716054799440/t0(0) o38->soaked-MDT0002-lwp-OST0011@192.168.1.110@o2ib:12/10 lens 520/544 e 0 to 1 dl 1513255140 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [45019.467078] Lustre: 5693:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [45019.606384] [ 3971] 0 3971 27511 156 12 32 0 agetty [45019.633815] [ 3975] 0 3975 27511 160 10 33 0 agetty [45019.661254] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45019.691779] [151572] 0 151572 610 33 4 9 0 systemd-cgroups [45019.722271] [151574] 0 151574 610 26 4 24 0 systemd-cgroups [45019.752754] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45019.783210] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45019.813645] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45019.844037] Out of memory: Kill process 3689 (munged) score 0 or sacrifice child [45019.869266] Killed process 3689 (munged) total-vm:96844kB, anon-rss:0kB, file-rss:656kB, shmem-rss:0kB [45019.942098] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45019.969885] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45019.986728] CPU: 20 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45020.026095] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45020.063821] ffff880805c80000 0000000035bd910c ffff880805c8b998 ffffffff816a3e2d [45020.089001] ffff880805c8ba28 ffffffff8169f246 ffffffff810e93ac ffff8803fffb8e50 [45020.114168] ffff8803fffb8e68 0000000000000202 ffff880805c8ba80 ffff880805c8ba18 [45020.139309] Call Trace: [45020.148071] [] dump_stack+0x19/0x1b [45020.165695] [] dump_header+0x90/0x229 [45020.183909] [] ? ktime_get_ts64+0x4c/0xf0 [45020.203263] [] oom_kill_process+0x254/0x3d0 [45020.223193] [] ? oom_unkillable_task+0xcd/0x120 [45020.244262] [] ? find_lock_task_mm+0x56/0xc0 [45020.264473] [] out_of_memory+0x4b6/0x4f0 [45020.283536] [] __alloc_pages_slowpath+0x5d6/0x724 [45020.305163] [] __alloc_pages_nodemask+0x405/0x420 [45020.326795] [] __vmalloc_node_range+0x12f/0x270 [45020.347911] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45020.371265] [] vzalloc_node+0x47/0x50 [45020.389547] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45020.412939] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45020.435746] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45020.459120] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45020.480495] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45020.505565] [] kthread+0xcf/0xe0 [45020.522339] [] ? insert_kthread_work+0x40/0x40 [45020.543119] [] ret_from_fork+0x58/0x90 [45020.561611] [] ? insert_kthread_work+0x40/0x40 [45020.582392] Mem-Info: [45020.590585] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234747 mapped:1547 shmem:0 pagetables:422 bounce:0 free:39367 free_pcp:0 free_cma:0 [45020.700777] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45020.839897] lowmem_reserve[]: 0 2580 15620 15620 [45020.855906] Node 0 DMA32 free:59460kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:176kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:372 all_unreclaimable? yes [45021.004954] lowmem_reserve[]: 0 0 13040 13040 [45021.020217] Node 0 Normal free:36860kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311788kB kernel_stack:10624kB pagetables:656kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:216 all_unreclaimable? yes [45021.174401] lowmem_reserve[]: 0 0 0 0 [45021.187380] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45021.232378] Node 0 DMA32: 441*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59460kB [45021.288912] Node 0 Normal: 213*4kB (UEM) 323*8kB (UEM) 577*16kB (UEM) 474*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36860kB [45021.339772] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45021.369412] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45021.398184] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45021.427810] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45021.456567] 1791 total pagecache pages [45021.469877] 0 pages in swap cache [45021.481758] Swap cache stats: add 71108, delete 71108, find 15055/20478 [45021.504525] Free swap = 16267708kB [45021.516982] Total swap = 16319484kB [45021.529433] 8369063 pages RAM [45021.540160] 0 pages HighMem/MovableOnly [45021.553744] 241261 pages reserved [45021.565607] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45021.592376] [ 1034] 0 1034 29712 215 27 85 0 lvmetad [45021.619996] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45021.649332] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [45021.677820] [ 2539] 0 2539 13863 113 26 137 -1000 auditd [45021.705156] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45021.733921] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45021.761249] [ 2955] 0 2955 6051 217 16 88 0 systemd-logind [45021.790866] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45021.818191] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45021.844938] [ 3883] 0 3883 6464 163 19 52 0 atd [45021.871396] [ 3971] 0 3971 27511 156 12 32 0 agetty [45021.898706] [ 3975] 0 3975 27511 160 10 33 0 agetty [45021.926058] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45021.956505] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45021.986924] [151574] 0 151574 610 26 4 24 0 systemd-cgroups [45022.017312] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45022.047671] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45022.078006] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45022.108317] Out of memory: Kill process 1034 (lvmetad) score 0 or sacrifice child [45022.133754] Killed process 1034 (lvmetad) total-vm:118848kB, anon-rss:0kB, file-rss:860kB, shmem-rss:0kB [45022.288874] systemd-logind invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45022.316980] systemd-logind cpuset=/ mems_allowed=0-1 [45022.334212] CPU: 11 PID: 2955 Comm: systemd-logind Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45022.373466] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45022.411278] ffff88041876bf40 0000000068fac475 ffff88042ce7f988 ffffffff816a3e2d [45022.436549] ffff88042ce7fa18 ffffffff8169f246 ffffffff810e93ac ffff880036accf30 [45022.461797] ffff880036accf48 0000000000000206 0000000000000206 ffff88042ce7fa08 [45022.487019] Call Trace: [45022.495880] [] dump_stack+0x19/0x1b [45022.513595] [] dump_header+0x90/0x229 [45022.531860] [] ? ktime_get_ts64+0x4c/0xf0 [45022.551244] [] ? delayacct_end+0x8f/0xb0 [45022.570312] [] oom_kill_process+0x254/0x3d0 [45022.590220] [] ? oom_unkillable_task+0xcd/0x120 [45022.611258] [] ? find_lock_task_mm+0x56/0xc0 [45022.631445] [] out_of_memory+0x4b6/0x4f0 [45022.650491] [] __alloc_pages_slowpath+0x5d6/0x724 [45022.672124] [] __alloc_pages_nodemask+0x405/0x420 [45022.693743] [] alloc_pages_vma+0xb5/0x200 [45022.713069] [] read_swap_cache_async+0xed/0x160 [45022.734109] [] ? mem_cgroup_update_page_stat+0x16/0x50 [45022.757153] [] swapin_readahead+0xa8/0x110 [45022.776767] [] handle_mm_fault+0xadb/0xfa0 [45022.796391] [] __do_page_fault+0x154/0x450 [45022.815998] [] do_page_fault+0x35/0x90 [45022.834465] [] page_fault+0x28/0x30 [45022.852070] Mem-Info: [45022.860227] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:135 inactive_file:111 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234745 mapped:1547 shmem:0 pagetables:395 bounce:0 free:39404 free_pcp:0 free_cma:0 [45022.970355] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45023.109685] lowmem_reserve[]: 0 2580 15620 15620 [45023.125750] Node 0 DMA32 free:59460kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:176kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1380 all_unreclaimable? yes [45023.275338] lowmem_reserve[]: 0 0 13040 13040 [45023.290702] Node 0 Normal free:37004kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:212kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311780kB kernel_stack:10624kB pagetables:552kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5490 all_unreclaimable? yes [45023.445509] lowmem_reserve[]: 0 0 0 0 [45023.458626] Node 1 Normal free:45576kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567436kB kernel_stack:18864kB pagetables:852kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1299 all_unreclaimable? yes [45023.614655] lowmem_reserve[]: 0 0 0 0 [45023.627806] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45023.673193] Node 0 DMA32: 441*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59460kB [45023.730175] Node 0 Normal: 243*4kB (UEM) 326*8kB (UEM) 577*16kB (UEM) 474*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37004kB [45023.781469] Node 1 Normal: 1068*4kB (UEM) 727*8kB (UEM) 368*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45576kB [45023.833427] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45023.863311] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45023.892332] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45023.922079] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45023.950958] 1791 total pagecache pages [45023.964389] 0 pages in swap cache [45023.976381] Swap cache stats: add 71116, delete 71116, find 15056/20480 [45023.999261] Free swap = 16268048kB [45024.011840] Total swap = 16319484kB [45024.024434] 8369063 pages RAM [45024.035290] 0 pages HighMem/MovableOnly [45024.049006] 241261 pages reserved [45024.061003] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45024.087904] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45024.117379] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [45024.145998] [ 2539] 0 2539 13863 113 26 137 -1000 auditd [45024.173477] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45024.202376] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45024.229825] [ 2955] 0 2955 6051 217 16 88 0 systemd-logind [45024.259538] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45024.286950] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45024.313760] [ 3883] 0 3883 6464 163 19 52 0 atd [45024.340267] [ 3971] 0 3971 27511 156 12 32 0 agetty [45024.367611] [ 3975] 0 3975 27511 160 10 33 0 agetty [45024.394959] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45024.425408] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45024.455834] [151574] 0 151574 610 26 4 24 0 systemd-cgroups [45024.486227] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45024.516595] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45024.546977] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45024.577288] Out of memory: Kill process 2955 (systemd-logind) score 0 or sacrifice child [45024.604748] Killed process 2955 (systemd-logind) total-vm:24204kB, anon-rss:0kB, file-rss:868kB, shmem-rss:0kB [45024.667202] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45024.695504] systemd-cgroups cpuset=/ mems_allowed=0-1 [45024.713047] CPU: 0 PID: 151572 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45024.752922] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45024.790760] ffff880811c78fd0 0000000050754271 ffff8808243a3a00 ffffffff816a3e2d [45024.816062] ffff8808243a3a90 ffffffff8169f246 ffffffff810e93ac ffff8807d02399f0 [45024.841370] ffff8807d0239a08 0000000000000206 0000000000000206 ffff8808243a3a80 [45024.866662] Call Trace: [45024.875561] [] dump_stack+0x19/0x1b [45024.893309] [] dump_header+0x90/0x229 [45024.911605] [] ? ktime_get_ts64+0x4c/0xf0 [45024.931022] [] ? delayacct_end+0x8f/0xb0 [45024.950131] [] oom_kill_process+0x254/0x3d0 [45024.970070] [] ? oom_unkillable_task+0xcd/0x120 [45024.991130] [] ? find_lock_task_mm+0x56/0xc0 [45025.011322] [] out_of_memory+0x4b6/0x4f0 [45025.030366] [] __alloc_pages_slowpath+0x5d6/0x724 [45025.051998] [] __alloc_pages_nodemask+0x405/0x420 [45025.073641] [] alloc_pages_vma+0xb5/0x200 [45025.092979] [] ? anon_vma_interval_tree_insert+0x8b/0x90 [45025.116614] [] do_cow_fault+0x66/0x290 [45025.135065] [] handle_mm_fault+0x2bd/0xfa0 [45025.154686] [] __do_page_fault+0x154/0x450 [45025.174307] [] do_page_fault+0x35/0x90 [45025.192760] [] page_fault+0x28/0x30 [45025.210370] Mem-Info: [45025.218540] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:121 inactive_file:125 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234745 mapped:1547 shmem:0 pagetables:395 bounce:0 free:39404 free_pcp:116 free_cma:0 [45025.329209] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45025.468515] lowmem_reserve[]: 0 2580 15620 15620 [45025.484589] Node 0 DMA32 free:59460kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:176kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2556 all_unreclaimable? yes [45025.634120] lowmem_reserve[]: 0 0 13040 13040 [45025.649453] Node 0 Normal free:36880kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:156kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311780kB kernel_stack:10624kB pagetables:492kB unstable:0kB bounce:0kB free_pcp:208kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5490 all_unreclaimable? yes [45025.804884] lowmem_reserve[]: 0 0 0 0 [45025.818016] Node 1 Normal free:45328kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567432kB kernel_stack:18864kB pagetables:848kB unstable:0kB bounce:0kB free_pcp:256kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1306 all_unreclaimable? yes [45025.974702] lowmem_reserve[]: 0 0 0 0 [45025.987873] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45026.033235] Node 0 DMA32: 441*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59460kB [45026.090195] Node 0 Normal: 212*4kB (UEM) 326*8kB (UEM) 577*16kB (UEM) 474*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36880kB [45026.141492] Node 1 Normal: 1014*4kB (UEM) 723*8kB (UEM) 368*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45328kB [45026.193473] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45026.223336] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45026.252371] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45026.282246] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45026.311231] 1791 total pagecache pages [45026.324789] 0 pages in swap cache [45026.336895] Swap cache stats: add 71118, delete 71118, find 15058/20482 [45026.359888] Free swap = 16268368kB [45026.372592] Total swap = 16319484kB [45026.385282] 8369063 pages RAM [45026.396245] 0 pages HighMem/MovableOnly [45026.410074] 241261 pages reserved [45026.422177] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45026.449184] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45026.478792] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [45026.507558] [ 2539] 0 2539 13863 113 26 137 -1000 auditd [45026.535177] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45026.564221] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45026.591800] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45026.619350] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45026.646293] [ 3883] 0 3883 6464 163 19 52 0 atd [45026.672915] [ 3971] 0 3971 27511 156 12 32 0 agetty [45026.700378] [ 3975] 0 3975 27511 160 10 33 0 agetty [45026.727844] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45026.758414] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45026.788958] [151574] 0 151574 610 26 4 24 0 systemd-cgroups [45026.819491] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45026.849985] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45026.880451] [151578] 0 151578 79 0 4 9 0 systemd-cgroups [45026.910877] Out of memory: Kill process 3883 (atd) score 0 or sacrifice child [45026.935290] Killed process 3883 (atd) total-vm:25856kB, anon-rss:0kB, file-rss:652kB, shmem-rss:0kB [45026.990350] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45027.018149] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45027.035058] CPU: 19 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45027.074630] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45027.112483] ffff880805c80000 0000000035bd910c ffff880805c8b998 ffffffff816a3e2d [45027.137667] ffff880805c8ba28 ffffffff8169f246 ffffffff810e93ac ffff8803fffb8e50 [45027.162853] ffff8803fffb8e68 0000000000000206 0000000000000206 ffff880805c8ba18 [45027.188038] Call Trace: [45027.196851] [] dump_stack+0x19/0x1b [45027.214494] [] dump_header+0x90/0x229 [45027.232709] [] ? ktime_get_ts64+0x4c/0xf0 [45027.252050] [] ? delayacct_end+0x8f/0xb0 [45027.271082] [] oom_kill_process+0x254/0x3d0 [45027.290950] [] ? oom_unkillable_task+0xcd/0x120 [45027.311947] [] ? find_lock_task_mm+0x56/0xc0 [45027.332066] [] out_of_memory+0x4b6/0x4f0 [45027.351030] [] __alloc_pages_slowpath+0x5d6/0x724 [45027.372565] [] __alloc_pages_nodemask+0x405/0x420 [45027.394106] [] __vmalloc_node_range+0x12f/0x270 [45027.415116] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45027.438372] [] vzalloc_node+0x47/0x50 [45027.456506] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45027.479791] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45027.502498] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45027.525784] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45027.547072] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45027.572057] [] kthread+0xcf/0xe0 [45027.588749] [] ? insert_kthread_work+0x40/0x40 [45027.609441] [] ret_from_fork+0x58/0x90 [45027.627835] [] ? insert_kthread_work+0x40/0x40 [45027.648550] Mem-Info: [45027.656645] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:114 inactive_file:132 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234742 mapped:1547 shmem:0 pagetables:360 bounce:0 free:39353 free_pcp:121 free_cma:0 [45027.767001] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45027.906133] lowmem_reserve[]: 0 2580 15620 15620 [45027.922147] Node 0 DMA32 free:59464kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:176kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:674 all_unreclaimable? yes [45028.071182] lowmem_reserve[]: 0 0 13040 13040 [45028.086415] Node 0 Normal free:36788kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:128kB inactive_file:28kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311772kB kernel_stack:10624kB pagetables:416kB unstable:0kB bounce:0kB free_pcp:484kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:704 all_unreclaimable? yes [45028.241105] lowmem_reserve[]: 0 0 0 0 [45028.254065] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45028.299042] Node 0 DMA32: 442*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59464kB [45028.355604] Node 0 Normal: 96*4kB (UEM) 328*8kB (UEM) 578*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36416kB [45028.406176] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45028.435816] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45028.464655] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45028.494279] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45028.523039] 1794 total pagecache pages [45028.536365] 2 pages in swap cache [45028.548243] Swap cache stats: add 71134, delete 71132, find 15063/20492 [45028.571004] Free swap = 16268576kB [45028.583462] Total swap = 16319484kB [45028.595923] 8369063 pages RAM [45028.606660] 0 pages HighMem/MovableOnly [45028.620245] 241261 pages reserved [45028.632111] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45028.658884] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45028.688228] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [45028.716717] [ 2539] 0 2539 13863 113 26 137 -1000 auditd [45028.744061] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45028.772835] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45028.800175] [ 3694] 0 3694 6779 148 18 62 0 xinetd [45028.827507] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45028.854260] [ 3971] 0 3971 27511 156 12 32 0 agetty [45028.881578] [ 3975] 0 3975 27511 160 10 33 0 agetty [45028.908930] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45028.939402] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45028.969860] [151574] 0 151574 610 26 4 25 0 systemd-cgroups [45029.000308] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45029.030745] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45029.061143] [151578] 0 151578 79 0 4 10 0 systemd-cgroups [45029.091510] Out of memory: Kill process 3694 (xinetd) score 0 or sacrifice child [45029.116761] Killed process 3694 (xinetd) total-vm:27116kB, anon-rss:0kB, file-rss:592kB, shmem-rss:0kB [45029.258122] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45029.286303] systemd-cgroups cpuset=/ mems_allowed=0-1 [45029.303867] CPU: 1 PID: 151571 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45029.343770] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45029.381621] ffff880804038000 0000000075ff9a13 ffff88005e1db988 ffffffff816a3e2d [45029.406928] ffff88005e1dba18 ffffffff8169f246 ffffffff810e93ac ffff880409c05a60 [45029.432194] ffff880409c05a78 0000000000000202 0000000000000202 ffff88005e1dba08 [45029.457496] Call Trace: [45029.466410] [] dump_stack+0x19/0x1b [45029.484180] [] dump_header+0x90/0x229 [45029.502497] [] ? ktime_get_ts64+0x4c/0xf0 [45029.521937] [] ? delayacct_end+0x8f/0xb0 [45029.541073] [] oom_kill_process+0x254/0x3d0 [45029.561042] [] ? oom_unkillable_task+0xcd/0x120 [45029.582130] [] ? find_lock_task_mm+0x56/0xc0 [45029.602334] [] out_of_memory+0x4b6/0x4f0 [45029.621371] [] __alloc_pages_slowpath+0x5d6/0x724 [45029.642973] [] __alloc_pages_nodemask+0x405/0x420 [45029.664572] [] alloc_pages_vma+0xb5/0x200 [45029.683891] [] read_swap_cache_async+0xed/0x160 [45029.704957] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [45029.727429] [] swapin_readahead+0xa8/0x110 [45029.747035] [] handle_mm_fault+0xadb/0xfa0 [45029.766650] [] __do_page_fault+0x154/0x450 [45029.786255] [] do_page_fault+0x35/0x90 [45029.804713] [] page_fault+0x28/0x30 [45029.822314] Mem-Info: [45029.830465] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:156 inactive_file:90 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234739 mapped:1548 shmem:0 pagetables:342 bounce:0 free:39410 free_pcp:0 free_cma:0 [45029.940280] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45030.079596] lowmem_reserve[]: 0 2580 15620 15620 [45030.095656] Node 0 DMA32 free:59464kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:176kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:2154 all_unreclaimable? yes [45030.245136] lowmem_reserve[]: 0 0 13040 13040 [45030.260459] Node 0 Normal free:37004kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:296kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3796kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311772kB kernel_stack:10624kB pagetables:344kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:5453 all_unreclaimable? yes [45030.415690] lowmem_reserve[]: 0 0 0 0 [45030.428802] Node 1 Normal free:45596kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567420kB kernel_stack:18864kB pagetables:848kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1422 all_unreclaimable? yes [45030.584697] lowmem_reserve[]: 0 0 0 0 [45030.597818] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45030.643130] Node 0 DMA32: 442*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59464kB [45030.700072] Node 0 Normal: 210*4kB (UEM) 329*8kB (UEM) 578*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36880kB [45030.751327] Node 1 Normal: 1071*4kB (UEM) 728*8kB (UEM) 368*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45596kB [45030.803248] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45030.833131] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45030.862140] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45030.892012] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45030.921009] 1792 total pagecache pages [45030.934566] 0 pages in swap cache [45030.946674] Swap cache stats: add 71150, delete 71150, find 15064/20495 [45030.969671] Free swap = 16268824kB [45030.982377] Total swap = 16319484kB [45030.995084] 8369063 pages RAM [45031.006063] 0 pages HighMem/MovableOnly [45031.019899] 241261 pages reserved [45031.032012] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45031.059031] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45031.088626] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [45031.117369] [ 2539] 0 2539 13863 113 26 137 -1000 auditd [45031.144964] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45031.173985] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45031.201562] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45031.228531] [ 3971] 0 3971 27511 156 12 32 0 agetty [45031.256047] [ 3975] 0 3975 27511 160 10 33 0 agetty [45031.283573] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45031.314181] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45031.344789] [151574] 0 151574 610 26 4 25 0 systemd-cgroups [45031.375345] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45031.405873] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45031.436392] [151578] 0 151578 79 0 4 10 0 systemd-cgroups [45031.466844] Out of memory: Kill process 3975 (agetty) score 0 or sacrifice child [45031.492137] Killed process 3975 (agetty) total-vm:110044kB, anon-rss:0kB, file-rss:640kB, shmem-rss:0kB [45031.525539] systemd-cgroups invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0 [45031.553942] systemd-cgroups cpuset=/ mems_allowed=0-1 [45031.571645] CPU: 4 PID: 151577 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45031.611937] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45031.649969] ffff880811c7bf40 00000000dc9421c4 ffff8807c54f3920 ffffffff816a3e2d [45031.675577] ffff8807c54f39b0 ffffffff8169f246 ffffffff810e93ac ffff8807d0239980 [45031.701130] ffff8807d0239998 0000000000000206 0000000000000206 ffff8807c54f39a0 [45031.726675] Call Trace: [45031.735793] [] dump_stack+0x19/0x1b [45031.753823] [] dump_header+0x90/0x229 [45031.772394] [] ? ktime_get_ts64+0x4c/0xf0 [45031.792107] [] ? delayacct_end+0x8f/0xb0 [45031.811392] [] oom_kill_process+0x254/0x3d0 [45031.831524] [] ? oom_unkillable_task+0xcd/0x120 [45031.852818] [] ? find_lock_task_mm+0x56/0xc0 [45031.873261] [] out_of_memory+0x4b6/0x4f0 [45031.892538] [] __alloc_pages_slowpath+0x5d6/0x724 [45031.914278] [] __alloc_pages_nodemask+0x405/0x420 [45031.936024] [] alloc_pages_current+0x98/0x110 [45031.956590] [] __page_cache_alloc+0x97/0xb0 [45031.976396] [] filemap_fault+0x170/0x410 [45031.995386] [] ext4_filemap_fault+0x36/0x50 [ext4] [45032.017219] [] __do_fault+0x52/0xe0 [45032.034762] [] do_read_fault.isra.44+0x4b/0x130 [45032.055736] [] handle_mm_fault+0x691/0xfa0 [45032.075273] [] ? change_protection+0x65/0xa0 [45032.095385] [] __do_page_fault+0x154/0x450 [45032.114928] [] do_page_fault+0x35/0x90 [45032.133339] [] page_fault+0x28/0x30 [45032.150881] Mem-Info: [45032.158971] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:93 inactive_file:154 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234739 mapped:1548 shmem:0 pagetables:342 bounce:0 free:39409 free_pcp:48 free_cma:0 [45032.268659] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45032.407693] lowmem_reserve[]: 0 2580 15620 15620 [45032.423679] Node 0 DMA32 free:59464kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:176kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3634 all_unreclaimable? yes [45032.572924] lowmem_reserve[]: 0 0 13040 13040 [45032.588147] Node 0 Normal free:37000kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:44kB inactive_file:116kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3796kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311772kB kernel_stack:10624kB pagetables:344kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:445 all_unreclaimable? yes [45032.742198] lowmem_reserve[]: 0 0 0 0 [45032.755114] Node 1 Normal free:45472kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567420kB kernel_stack:18864kB pagetables:808kB unstable:0kB bounce:0kB free_pcp:316kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1422 all_unreclaimable? yes [45032.911069] lowmem_reserve[]: 0 0 0 0 [45032.924105] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45032.969252] Node 0 DMA32: 442*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59464kB [45033.025958] Node 0 Normal: 240*4kB (UEM) 329*8kB (UEM) 578*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37000kB [45033.076985] Node 1 Normal: 1045*4kB (UEM) 722*8kB (EM) 368*16kB (UEM) 324*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45348kB [45033.128373] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45033.158135] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45033.187032] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45033.216779] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45033.245654] 1792 total pagecache pages [45033.259084] 0 pages in swap cache [45033.271078] Swap cache stats: add 71150, delete 71150, find 15064/20495 [45033.293959] Free swap = 16268956kB [45033.306511] Total swap = 16319484kB [45033.319090] 8369063 pages RAM [45033.329945] 0 pages HighMem/MovableOnly [45033.343658] 241261 pages reserved [45033.355653] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45033.382548] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45033.412021] [ 1081] 0 1081 1588718 4024 133 0 -1000 multipathd [45033.440648] [ 2539] 0 2539 13863 113 26 137 -1000 auditd [45033.468114] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45033.497060] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45033.524505] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45033.551358] [ 3971] 0 3971 27511 156 12 32 0 agetty [45033.578790] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45033.609336] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45033.639866] [151574] 0 151574 610 26 4 25 0 systemd-cgroups [45033.670442] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45033.700959] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45033.731392] [151578] 0 151578 79 0 4 10 0 systemd-cgroups [45033.761854] Out of memory: Kill process 3971 (agetty) score 0 or sacrifice child [45033.787155] Killed process 3971 (agetty) total-vm:110044kB, anon-rss:0kB, file-rss:624kB, shmem-rss:0kB [45033.823386] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45033.851304] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45033.868313] CPU: 2 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45033.907662] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45033.945603] ffff880805c80000 0000000035bd910c ffff880805c8b998 ffffffff816a3e2d [45033.970968] ffff880805c8ba28 ffffffff8169f246 ffffffff810e93ac ffff8803fffb8e50 [45033.996334] ffff8803fffb8e68 0000000000000202 0000000000000202 ffff880805c8ba18 [45034.021723] Call Trace: [45034.030673] [] dump_stack+0x19/0x1b [45034.048470] [] dump_header+0x90/0x229 [45034.066915] [] ? ktime_get_ts64+0x4c/0xf0 [45034.086386] [] ? delayacct_end+0x8f/0xb0 [45034.105592] [] oom_kill_process+0x254/0x3d0 [45034.125672] [] ? oom_unkillable_task+0xcd/0x120 [45034.146779] [] ? find_lock_task_mm+0x56/0xc0 [45034.167049] [] out_of_memory+0x4b6/0x4f0 [45034.186154] [] __alloc_pages_slowpath+0x5d6/0x724 [45034.207801] [] __alloc_pages_nodemask+0x405/0x420 [45034.229433] [] __vmalloc_node_range+0x12f/0x270 [45034.250458] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45034.273776] [] vzalloc_node+0x47/0x50 [45034.292023] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45034.315434] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45034.338201] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45034.361605] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45034.382964] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45034.408016] [] kthread+0xcf/0xe0 [45034.424794] [] ? insert_kthread_work+0x40/0x40 [45034.445567] [] ret_from_fork+0x58/0x90 [45034.464073] [] ? insert_kthread_work+0x40/0x40 [45034.484882] Mem-Info: [45034.493029] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:62 inactive_file:184 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234738 mapped:1547 shmem:0 pagetables:320 bounce:0 free:39396 free_pcp:20 free_cma:0 [45034.603353] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45034.742790] lowmem_reserve[]: 0 2580 15620 15620 [45034.758870] Node 0 DMA32 free:59464kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:128kB unstable:0kB bounce:0kB free_pcp:80kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4830 all_unreclaimable? yes [45034.908905] lowmem_reserve[]: 0 0 13040 13040 [45034.924242] Node 0 Normal free:37004kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:48kB inactive_file:108kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311772kB kernel_stack:10624kB pagetables:344kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:401 all_unreclaimable? yes [45035.078994] lowmem_reserve[]: 0 0 0 0 [45035.092132] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45035.137327] Node 0 DMA32: 442*4kB (UEM) 718*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59464kB [45035.194119] Node 0 Normal: 241*4kB (UEM) 329*8kB (UEM) 578*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37004kB [45035.245331] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45035.275127] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45035.304025] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45035.333767] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45035.362624] 1791 total pagecache pages [45035.376039] 0 pages in swap cache [45035.387997] Swap cache stats: add 71150, delete 71150, find 15064/20495 [45035.410864] Free swap = 16269084kB [45035.423448] Total swap = 16319484kB [45035.436020] 8369063 pages RAM [45035.446914] 0 pages HighMem/MovableOnly [45035.460555] 241261 pages reserved [45035.472510] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45035.499379] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45035.528901] [ 1081] 0 1081 1588718 4023 133 0 -1000 multipathd [45035.557605] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45035.585057] [ 2588] 81 2588 9199 204 21 139 -900 dbus-daemon [45035.613969] [ 2642] 0 2642 1770 27 8 37 0 mcelog [45035.641389] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45035.668251] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45035.698828] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45035.729424] [151574] 0 151574 610 26 4 25 0 systemd-cgroups [45035.760054] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45035.790598] [151577] 0 151577 79 0 4 6 0 systemd-cgroups [45035.821153] [151578] 0 151578 79 0 4 10 0 systemd-cgroups [45035.851681] Out of memory: Kill process 2642 (mcelog) score 0 or sacrifice child [45035.877067] Killed process 2642 (mcelog) total-vm:7080kB, anon-rss:0kB, file-rss:108kB, shmem-rss:0kB [45035.971170] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45035.999529] systemd-cgroups cpuset=/ mems_allowed=0-1 [45036.017246] CPU: 1 PID: 151571 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45036.057562] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45036.095639] ffff880804038000 0000000075ff9a13 ffff88005e1db988 ffffffff816a3e2d [45036.121175] ffff88005e1dba18 ffffffff8169f246 ffffffff810e93ac ffff880409c05a60 [45036.146660] ffff880409c05a78 0000000000000206 ffff88005e1dba70 ffff88005e1dba08 [45036.172139] Call Trace: [45036.181244] [] dump_stack+0x19/0x1b [45036.199193] [] dump_header+0x90/0x229 [45036.217661] [] ? ktime_get_ts64+0x4c/0xf0 [45036.237274] [] oom_kill_process+0x254/0x3d0 [45036.257434] [] ? oom_unkillable_task+0xcd/0x120 [45036.278709] [] ? find_lock_task_mm+0x56/0xc0 [45036.299104] [] out_of_memory+0x4b6/0x4f0 [45036.318351] [] __alloc_pages_slowpath+0x5d6/0x724 [45036.340130] [] __alloc_pages_nodemask+0x405/0x420 [45036.361894] [] alloc_pages_vma+0xb5/0x200 [45036.381365] [] read_swap_cache_async+0xed/0x160 [45036.402507] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [45036.425055] [] swapin_readahead+0xa8/0x110 [45036.444750] [] handle_mm_fault+0xadb/0xfa0 [45036.464453] [] __do_page_fault+0x154/0x450 [45036.484167] [] do_page_fault+0x35/0x90 [45036.502724] [] page_fault+0x28/0x30 [45036.520408] Mem-Info: [45036.528673] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:123 inactive_file:123 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:312 bounce:0 free:39385 free_pcp:60 free_cma:0 [45036.637774] LNet: 5683:0:(o2iblnd_cb.c:3192:kiblnd_check_conns()) Timed out tx for 192.168.1.110@o2ib: 5 seconds [45036.674036] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45036.813636] lowmem_reserve[]: 0 2580 15620 15620 [45036.829725] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:16kB inactive_file:0kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:96kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1796 all_unreclaimable? no [45036.979434] lowmem_reserve[]: 0 0 13040 13040 [45036.994916] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:176kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:344kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:312 all_unreclaimable? yes [45037.150018] lowmem_reserve[]: 0 0 0 0 [45037.163256] Node 1 Normal free:45668kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:300kB inactive_file:512kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:2356kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18864kB pagetables:808kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1737 all_unreclaimable? yes [45037.262749] kworker/21:1: page allocation failure: order:8, mode:0x80d0 [45037.262751] CPU: 21 PID: 246 Comm: kworker/21:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45037.262758] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45037.262762] Workqueue: ib_cm cm_work_handler [ib_cm] [45037.262764] 00000000000080d0 00000000eeeeb502 ffff88042bc0f5a8 ffffffff816a3e2d [45037.262765] ffff88042bc0f638 ffffffff81188820 0000000000000000 00000000ffffffff [45037.262766] ffffffffffffff00 000080d000000000 ffff88042bc0f608 00000000eeeeb502 [45037.262766] Call Trace: [45037.262769] [] dump_stack+0x19/0x1b [45037.262771] [] warn_alloc_failed+0x110/0x180 [45037.262773] [] __alloc_pages_slowpath+0x6b6/0x724 [45037.262774] [] __alloc_pages_nodemask+0x405/0x420 [45037.262776] [] dma_generic_alloc_coherent+0x8f/0x140 [45037.262779] [] x86_swiotlb_alloc_coherent+0x21/0x50 [45037.262791] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [45037.262797] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [45037.262803] [] create_qp_common+0x536/0x1000 [mlx4_ib] [45037.262805] [] ? dma_pool_free+0xa7/0xd0 [45037.262810] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [45037.262813] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [45037.262818] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [45037.262826] [] ib_create_qp+0x7a/0x2f0 [ib_core] [45037.262829] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [45037.262834] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [45037.262842] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [45037.262847] [] kiblnd_passive_connect+0xa99/0x18c0 [ko2iblnd] [45037.262849] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [45037.262852] [] kiblnd_cm_callback+0x755/0x2380 [ko2iblnd] [45037.262855] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [45037.262857] [] cm_process_work+0x27/0x120 [ib_cm] [45037.262859] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [45037.262860] [] cm_work_handler+0x395/0x1306 [ib_cm] [45037.262862] [] ? pwq_activate_delayed_work+0x3f/0xb0 [45037.262863] [] process_one_work+0x17a/0x440 [45037.262865] [] worker_thread+0x126/0x3c0 [45037.262866] [] ? manage_workers.isra.24+0x2a0/0x2a0 [45037.262867] [] kthread+0xcf/0xe0 [45037.262868] [] ? insert_kthread_work+0x40/0x40 [45037.262870] [] ret_from_fork+0x58/0x90 [45037.262871] [] ? insert_kthread_work+0x40/0x40 [45037.262871] Mem-Info: [45037.262875] active_anon:0 inactive_anon:2 isolated_anon:0 active_file:130 inactive_file:116 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1676 shmem:0 pagetables:312 bounce:0 free:39404 free_pcp:0 free_cma:0 [45037.262879] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45037.262880] lowmem_reserve[]: 0 2580 15620 15620 [45037.262884] Node 0 DMA32 free:59364kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:44kB inactive_file:0kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:96kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [45037.262885] lowmem_reserve[]: 0 0 13040 13040 [45037.262888] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:176kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:344kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:312 all_unreclaimable? yes [45037.262889] lowmem_reserve[]: 0 0 0 0 [45037.262893] Node 1 Normal free:45668kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:300kB inactive_file:512kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:2336kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18864kB pagetables:808kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1737 all_unreclaimable? yes [45037.262893] lowmem_reserve[]: 0 0 0 0 [45037.262898] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45037.262905] Node 0 DMA32: 437*4kB (UEM) 719*8kB (UEM) 504*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59436kB [45037.262910] Node 0 Normal: 241*4kB (UEM) 330*8kB (UEM) 578*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37012kB [45037.262915] Node 1 Normal: 1089*4kB (UEM) 728*8kB (UEM) 368*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45668kB [45037.262916] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45037.262916] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45037.262917] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45037.262917] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45037.262918] 1794 total pagecache pages [45037.262918] 1 pages in swap cache [45037.262919] Swap cache stats: add 71176, delete 71175, find 15064/20498 [45037.262919] Free swap = 16269228kB [45037.262919] Total swap = 16319484kB [45037.262920] 8369063 pages RAM [45037.262920] 0 pages HighMem/MovableOnly [45037.262921] 241261 pages reserved [45037.263664] kworker/21:1: page allocation failure: order:8, mode:0x80d0 [45037.263665] CPU: 21 PID: 246 Comm: kworker/21:1 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45037.263666] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45037.263669] Workqueue: ib_cm cm_work_handler [ib_cm] [45037.263670] 00000000000080d0 00000000eeeeb502 ffff88042bc0f558 ffffffff816a3e2d [45037.263671] ffff88042bc0f5e8 ffffffff81188820 ffffffff8118b540 0000000000000000 [45037.263672] ffffffffffffff00 000080d000000000 ffff88042bc0f5b8 00000000eeeeb502 [45037.263672] Call Trace: [45037.263674] [] dump_stack+0x19/0x1b [45037.263676] [] warn_alloc_failed+0x110/0x180 [45037.263677] [] ? drain_pages+0xb0/0xb0 [45037.263679] [] __alloc_pages_slowpath+0x6b6/0x724 [45037.263680] [] __alloc_pages_nodemask+0x405/0x420 [45037.263682] [] alloc_pages_current+0x98/0x110 [45037.263683] [] __get_free_pages+0xe/0x40 [45037.263685] [] swiotlb_alloc_coherent+0x5e/0x150 [45037.263687] [] x86_swiotlb_alloc_coherent+0x41/0x50 [45037.263694] [] mlx4_buf_direct_alloc.isra.7+0xc4/0x180 [mlx4_core] [45037.263699] [] mlx4_buf_alloc+0x1bb/0x260 [mlx4_core] [45037.263704] [] create_qp_common+0x536/0x1000 [mlx4_ib] [45037.263705] [] ? dma_pool_free+0xa7/0xd0 [45037.263710] [] mlx4_ib_create_qp+0x3b1/0xdc0 [mlx4_ib] [45037.263713] [] ? mlx4_ib_create_cq+0x2d2/0x430 [mlx4_ib] [45037.263718] [] mlx4_ib_create_qp_wrp+0x10/0x20 [mlx4_ib] [45037.263723] [] ib_create_qp+0x7a/0x2f0 [ib_core] [45037.263725] [] rdma_create_qp+0x34/0xb0 [rdma_cm] [45037.263729] [] kiblnd_create_conn+0xbf9/0x1960 [ko2iblnd] [45037.263734] [] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [45037.263738] [] kiblnd_passive_connect+0xa99/0x18c0 [ko2iblnd] [45037.263740] [] ? _cma_attach_to_dev+0x5c/0x70 [rdma_cm] [45037.263743] [] kiblnd_cm_callback+0x755/0x2380 [ko2iblnd] [45037.263745] [] cma_req_handler+0x1c6/0x490 [rdma_cm] [45037.263747] [] cm_process_work+0x27/0x120 [ib_cm] [45037.263749] [] cm_req_handler+0xb0b/0xe30 [ib_cm] [45037.263751] [] cm_work_handler+0x395/0x1306 [ib_cm] [45037.263759] [] ? pwq_activate_delayed_work+0x3f/0xb0 [45037.263760] [] process_one_work+0x17a/0x440 [45037.263761] [] worker_thread+0x126/0x3c0 [45037.263763] [] ? manage_workers.isra.24+0x2a0/0x2a0 [45037.263764] [] kthread+0xcf/0xe0 [45037.263765] [] ? insert_kthread_work+0x40/0x40 [45037.263766] [] ret_from_fork+0x58/0x90 [45037.263767] [] ? insert_kthread_work+0x40/0x40 [45037.263768] Mem-Info: [45037.263771] active_anon:0 inactive_anon:2 isolated_anon:0 active_file:130 inactive_file:116 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1676 shmem:0 pagetables:312 bounce:0 free:39404 free_pcp:0 free_cma:0 [45037.263775] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45037.263776] lowmem_reserve[]: 0 2580 15620 15620 [45037.263779] Node 0 DMA32 free:59364kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:44kB inactive_file:0kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:96kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [45037.263780] lowmem_reserve[]: 0 0 13040 13040 [45037.263783] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:176kB inactive_file:0kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:344kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:312 all_unreclaimable? yes [45037.263784] lowmem_reserve[]: 0 0 0 0 [45037.263787] Node 1 Normal free:45668kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:300kB inactive_file:512kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:2336kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18864kB pagetables:808kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1737 all_unreclaimable? yes [45037.263788] lowmem_reserve[]: 0 0 0 0 [45037.263793] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45037.263798] Node 0 DMA32: 437*4kB (UEM) 719*8kB (UEM) 504*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59436kB [45037.263802] Node 0 Normal: 241*4kB (UEM) 330*8kB (UEM) 578*16kB (UEM) 473*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37012kB [45037.263806] Node 1 Normal: 1089*4kB (UEM) 728*8kB (UEM) 368*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45668kB [45037.263807] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45037.263808] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45037.263808] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45037.263809] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45037.263809] 1794 total pagecache pages [45037.263809] 1 pages in swap cache [45037.263810] Swap cache stats: add 71176, delete 71175, find 15064/20498 [45037.263810] Free swap = 16269228kB [45037.263810] Total swap = 16319484kB [45037.263811] 8369063 pages RAM [45037.263811] 0 pages HighMem/MovableOnly [45037.263811] 241261 pages reserved [45041.625068] lowmem_reserve[]: 0 0 0 0 [45041.638452] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45041.684337] Node 0 DMA32: 437*4kB (UEM) 719*8kB (UEM) 504*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59436kB [45041.741834] Node 0 Normal: 241*4kB (UEM) 330*8kB (UEM) 578*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36980kB [45041.793579] Node 1 Normal: 1089*4kB (UEM) 728*8kB (UEM) 368*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45668kB [45041.845903] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45041.875945] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45041.905093] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45041.935080] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45041.964176] 1793 total pagecache pages [45041.977826] 1 pages in swap cache [45041.989983] Swap cache stats: add 71176, delete 71175, find 15064/20498 [45042.013036] Free swap = 16269228kB [45042.025762] Total swap = 16319484kB [45042.038453] 8369063 pages RAM [45042.049398] 0 pages HighMem/MovableOnly [45042.063180] 241261 pages reserved [45042.075222] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45042.102168] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45042.131642] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45042.160246] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45042.187677] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45042.216492] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45042.243368] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45042.273906] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45042.304431] [151574] 0 151574 610 26 4 25 0 systemd-cgroups [45042.334918] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45042.365360] [151577] 0 151577 79 0 4 7 0 systemd-cgroups [45042.395773] [151578] 0 151578 79 0 4 10 0 systemd-cgroups [45042.426148] Out of memory: Kill process 151574 (systemd-cgroups) score 0 or sacrifice child [45042.454511] Killed process 151574 (systemd-cgroups) total-vm:2440kB, anon-rss:0kB, file-rss:104kB, shmem-rss:0kB [45042.493810] systemd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45042.519551] systemd cpuset=/ mems_allowed=0-1 [45042.534706] CPU: 2 PID: 1 Comm: systemd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45042.569956] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45042.607706] ffff88017a210000 00000000f8626b8c ffff88017a21b988 ffffffff816a3e2d [45042.633041] ffff88017a21ba18 ffffffff8169f246 ffffffff810e93ac ffff88017a1271a0 [45042.658297] ffff88017a1271b8 0000000000000202 0000000000000202 ffff88017a21ba08 [45042.683474] Call Trace: [45042.692358] [] dump_stack+0x19/0x1b [45042.710266] [] dump_header+0x90/0x229 [45042.728598] [] ? ktime_get_ts64+0x4c/0xf0 [45042.748115] [] ? delayacct_end+0x8f/0xb0 [45042.767291] [] oom_kill_process+0x254/0x3d0 [45042.787495] [] ? oom_unkillable_task+0xcd/0x120 [45042.808704] [] ? find_lock_task_mm+0x56/0xc0 [45042.829002] [] out_of_memory+0x4b6/0x4f0 [45042.848255] [] __alloc_pages_slowpath+0x5d6/0x724 [45042.869932] [] __alloc_pages_nodemask+0x405/0x420 [45042.891653] [] alloc_pages_vma+0xb5/0x200 [45042.911017] [] ? end_swap_bio_write+0x80/0x80 [45042.931446] [] read_swap_cache_async+0xed/0x160 [45042.952423] [] swapin_readahead+0xa8/0x110 [45042.971982] [] handle_mm_fault+0xadb/0xfa0 [45042.991649] [] __do_page_fault+0x154/0x450 [45043.011226] [] do_page_fault+0x35/0x90 [45043.029793] [] page_fault+0x28/0x30 [45043.047314] Mem-Info: [45043.055395] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:99 inactive_file:147 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:312 bounce:0 free:39415 free_pcp:72 free_cma:0 [45043.165492] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45043.304805] lowmem_reserve[]: 0 2580 15620 15620 [45043.320838] Node 0 DMA32 free:59436kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1120kB pagetables:92kB unstable:0kB bounce:0kB free_pcp:12kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:6736 all_unreclaimable? yes [45043.470236] lowmem_reserve[]: 0 0 13040 13040 [45043.485676] Node 0 Normal free:36732kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:68kB inactive_file:88kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:276kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:316 all_unreclaimable? yes [45043.640762] lowmem_reserve[]: 0 0 0 0 [45043.653827] Node 1 Normal free:45668kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18864kB pagetables:808kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1240 all_unreclaimable? yes [45043.809583] lowmem_reserve[]: 0 0 0 0 [45043.822626] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45043.868025] Node 0 DMA32: 437*4kB (UEM) 719*8kB (UEM) 504*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59436kB [45043.924893] Node 0 Normal: 179*4kB (UEM) 330*8kB (UEM) 578*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36732kB [45043.976284] Node 1 Normal: 1089*4kB (UEM) 728*8kB (UEM) 368*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45668kB [45044.028456] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45044.058367] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45044.087367] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45044.117112] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45044.146215] 1791 total pagecache pages [45044.159907] 0 pages in swap cache [45044.172110] Swap cache stats: add 71180, delete 71180, find 15064/20498 [45044.194999] Free swap = 16269276kB [45044.207647] Total swap = 16319484kB [45044.220355] 8369063 pages RAM [45044.231477] 0 pages HighMem/MovableOnly [45044.245260] 241261 pages reserved [45044.257301] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45044.284204] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45044.313687] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45044.342315] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45044.369788] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45044.398694] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45044.425604] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45044.456163] [151572] 0 151572 610 26 4 9 0 systemd-cgroups [45044.486722] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45044.517255] [151577] 0 151577 79 0 4 7 0 systemd-cgroups [45044.547765] [151578] 0 151578 79 0 4 10 0 systemd-cgroups [45044.578253] Out of memory: Kill process 151572 (systemd-cgroups) score 0 or sacrifice child [45044.606723] Killed process 151572 (systemd-cgroups) total-vm:2440kB, anon-rss:0kB, file-rss:104kB, shmem-rss:0kB [45044.646815] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45044.674814] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45044.691921] CPU: 0 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45044.731550] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45044.769551] ffff880805c80000 0000000035bd910c ffff880805c8b998 ffffffff816a3e2d [45044.795020] ffff880805c8ba28 ffffffff8169f246 ffffffff810e93ac ffff8803fffb8e50 [45044.820465] ffff8803fffb8e68 0000000000000206 0000000000000206 ffff880805c8ba18 [45044.845916] Call Trace: [45044.854978] [] dump_stack+0x19/0x1b [45044.872896] [] dump_header+0x90/0x229 [45044.891352] [] ? ktime_get_ts64+0x4c/0xf0 [45044.910931] [] ? delayacct_end+0x8f/0xb0 [45044.930171] [] oom_kill_process+0x254/0x3d0 [45044.950290] [] ? oom_unkillable_task+0xcd/0x120 [45044.971533] [] ? find_lock_task_mm+0x56/0xc0 [45044.991894] [] out_of_memory+0x4b6/0x4f0 [45045.011088] [] __alloc_pages_slowpath+0x5d6/0x724 [45045.032840] [] __alloc_pages_nodemask+0x405/0x420 [45045.054559] [] __vmalloc_node_range+0x12f/0x270 [45045.075720] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45045.099125] [] vzalloc_node+0x47/0x50 [45045.117378] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45045.140771] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45045.163605] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45045.187010] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45045.208425] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45045.233523] [] kthread+0xcf/0xe0 [45045.250331] [] ? insert_kthread_work+0x40/0x40 [45045.271142] [] ret_from_fork+0x58/0x90 [45045.289666] [] ? insert_kthread_work+0x40/0x40 [45045.310479] Mem-Info: [45045.318699] active_anon:0 inactive_anon:0 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:304 bounce:0 free:39379 free_pcp:61 free_cma:0 [45045.429499] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45045.569056] lowmem_reserve[]: 0 2580 15620 15620 [45045.585178] Node 0 DMA32 free:59236kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:88kB unstable:0kB bounce:0kB free_pcp:244kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:536 all_unreclaimable? yes [45045.734944] lowmem_reserve[]: 0 0 13040 13040 [45045.750291] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:329 all_unreclaimable? yes [45045.904918] lowmem_reserve[]: 0 0 0 0 [45045.917990] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45045.963219] Node 0 DMA32: 383*4kB (UEM) 719*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59236kB [45046.020012] Node 0 Normal: 248*4kB (UEM) 330*8kB (UEM) 578*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37008kB [45046.071143] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45046.100858] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45046.129744] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45046.159477] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45046.188365] 1791 total pagecache pages [45046.201786] 1 pages in swap cache [45046.213778] Swap cache stats: add 71181, delete 71180, find 15064/20498 [45046.236682] Free swap = 16269312kB [45046.249258] Total swap = 16319484kB [45046.261824] 8369063 pages RAM [45046.272676] 0 pages HighMem/MovableOnly [45046.286366] 241261 pages reserved [45046.298339] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45046.325221] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45046.354667] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45046.383268] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45046.410714] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45046.439604] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45046.466509] [151571] 0 151571 79 0 4 21 0 systemd-cgroups [45046.497111] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45046.527669] [151577] 0 151577 79 0 4 7 0 systemd-cgroups [45046.558306] [151578] 0 151578 79 0 4 10 0 systemd-cgroups [45046.588883] Out of memory: Kill process 151571 (systemd-cgroups) score 0 or sacrifice child [45046.617444] Killed process 151571 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45046.661474] systemd-cgroups: page allocation failure: order:0, mode:0x200da [45046.661476] CPU: 3 PID: 151571 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45046.661477] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45046.661478] 00000000000200da 0000000075ff9a13 ffff88005e1dba98 ffffffff816a3e2d [45046.661479] ffff88005e1dbb28 ffffffff81188820 ffffffff8118b540 0000000000000000 [45046.661480] ffffffffffffffff 000200da00000000 ffff88005e1dbb28 0000000075ff9a13 [45046.661481] Call Trace: [45046.661483] [] dump_stack+0x19/0x1b [45046.661486] [] warn_alloc_failed+0x110/0x180 [45046.661487] [] ? drain_pages+0xb0/0xb0 [45046.661488] [] ? wake_up_atomic_t+0x30/0x30 [45046.661489] [] __alloc_pages_slowpath+0x6b6/0x724 [45046.661491] [] __alloc_pages_nodemask+0x405/0x420 [45046.661492] [] alloc_pages_vma+0xb5/0x200 [45046.661494] [] read_swap_cache_async+0xed/0x160 [45046.661504] [] ? ext4_filemap_fault+0x41/0x50 [ext4] [45046.661505] [] swapin_readahead+0xa8/0x110 [45046.661507] [] handle_mm_fault+0xadb/0xfa0 [45046.661509] [] __do_page_fault+0x154/0x450 [45046.661510] [] do_page_fault+0x35/0x90 [45046.661512] [] page_fault+0x28/0x30 [45046.661512] Mem-Info: [45046.661516] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:304 bounce:0 free:39439 free_pcp:0 free_cma:0 [45046.661520] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45046.661521] lowmem_reserve[]: 0 2580 15620 15620 [45046.661524] Node 0 DMA32 free:59476kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:4kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:88kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:15 all_unreclaimable? no [45046.661525] lowmem_reserve[]: 0 0 13040 13040 [45046.661529] Node 0 Normal free:37008kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:0kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:332kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:329 all_unreclaimable? yes [45046.661530] lowmem_reserve[]: 0 0 0 0 [45046.661533] Node 1 Normal free:45696kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18864kB pagetables:796kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1021 all_unreclaimable? no [45046.661534] lowmem_reserve[]: 0 0 0 0 [45046.661539] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45046.661544] Node 0 DMA32: 445*4kB (UEM) 719*8kB (UEM) 505*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59484kB [45046.661548] Node 0 Normal: 248*4kB (UEM) 330*8kB (UEM) 578*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37008kB [45046.661552] Node 1 Normal: 1097*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [45046.661553] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45046.661554] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45046.661554] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45046.661555] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45046.661555] 1792 total pagecache pages [45046.661555] 0 pages in swap cache [45046.661556] Swap cache stats: add 71181, delete 71181, find 15064/20498 [45046.661556] Free swap = 16269312kB [45046.661557] Total swap = 16319484kB [45046.661557] 8369063 pages RAM [45046.661558] 0 pages HighMem/MovableOnly [45046.661558] 241261 pages reserved [45047.324751] systemd-cgroups invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [45047.324752] systemd-cgroups cpuset=/ mems_allowed=0-1 [45047.324754] CPU: 19 PID: 151578 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.324754] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.324756] ffff880811c7af70 00000000b0554d0e ffff8807c54f7a70 ffffffff816a3e2d [45047.324757] ffff8807c54f7b00 ffffffff8169f246 ffffffff810e93ac ffff8807d0239670 [45047.324758] ffff8807d0239688 0000000000000202 0000000000000202 ffff8807c54f7af0 [45047.324758] Call Trace: [45047.324761] [] dump_stack+0x19/0x1b [45047.324762] [] dump_header+0x90/0x229 [45047.324764] [] ? ktime_get_ts64+0x4c/0xf0 [45047.324765] [] ? delayacct_end+0x8f/0xb0 [45047.324767] [] oom_kill_process+0x254/0x3d0 [45047.324768] [] ? oom_unkillable_task+0xcd/0x120 [45047.324769] [] ? find_lock_task_mm+0x56/0xc0 [45047.324771] [] out_of_memory+0x4b6/0x4f0 [45047.324772] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.324774] [] __alloc_pages_nodemask+0x405/0x420 [45047.324775] [] alloc_pages_vma+0xb5/0x200 [45047.324777] [] ? anon_vma_interval_tree_insert+0x8b/0x90 [45047.324778] [] handle_mm_fault+0xb60/0xfa0 [45047.324780] [] __do_page_fault+0x154/0x450 [45047.324781] [] do_page_fault+0x35/0x90 [45047.324783] [] page_fault+0x28/0x30 [45047.324783] Mem-Info: [45047.324788] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:0 free_cma:0 [45047.324791] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.324793] lowmem_reserve[]: 0 2580 15620 15620 [45047.324796] Node 0 DMA32 free:59444kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:86 all_unreclaimable? yes [45047.324797] lowmem_reserve[]: 0 0 13040 13040 [45047.324800] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:282 all_unreclaimable? yes [45047.324801] lowmem_reserve[]: 0 0 0 0 [45047.324805] Node 1 Normal free:45716kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18848kB pagetables:796kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1331 all_unreclaimable? yes [45047.324806] lowmem_reserve[]: 0 0 0 0 [45047.324811] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.324816] Node 0 DMA32: 452*4kB (UEM) 719*8kB (UEM) 500*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45047.324820] Node 0 Normal: 234*4kB (UEM) 321*8kB (EM) 576*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36848kB [45047.324824] Node 1 Normal: 1097*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [45047.324825] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.324826] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.324826] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.324827] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.324827] 1792 total pagecache pages [45047.324827] 8 pages in swap cache [45047.324828] Swap cache stats: add 71194, delete 71186, find 15067/20503 [45047.324828] Free swap = 16269316kB [45047.324829] Total swap = 16319484kB [45047.324829] 8369063 pages RAM [45047.324830] 0 pages HighMem/MovableOnly [45047.324830] 241261 pages reserved [45047.324830] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.324837] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.324839] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.324846] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.324847] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.324850] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.324874] [151576] 0 151576 79 0 4 13 0 systemd-cgroups [45047.324875] [151577] 0 151577 79 0 4 7 0 systemd-cgroups [45047.324875] [151578] 0 151578 79 0 4 13 0 systemd-cgroups [45047.324877] [151582] 0 151582 41 0 2 1 0 systemd-cgroups [45047.324877] [151583] 0 151583 41 0 2 1 0 systemd-cgroups [45047.324878] [151584] 0 151584 41 0 2 1 0 systemd-cgroups [45047.324879] [151585] 0 151585 41 0 2 1 0 systemd-cgroups [45047.324880] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.324881] Out of memory: Kill process 151576 (systemd-cgroups) score 0 or sacrifice child [45047.324882] Killed process 151576 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.331633] auditd invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=-1000 [45047.331635] auditd cpuset=/ mems_allowed=0-1 [45047.331637] CPU: 6 PID: 2539 Comm: auditd Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.331637] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.331640] ffff880819babf40 000000006189215e ffff88081d8db988 ffffffff816a3e2d [45047.331641] ffff88081d8dba18 ffffffff8169f246 ffffffff810e93ac ffff88082bc4a520 [45047.331642] ffff88082bc4a538 0000000000000206 0000000000000206 ffff88081d8dba08 [45047.331643] Call Trace: [45047.331646] [] dump_stack+0x19/0x1b [45047.331647] [] dump_header+0x90/0x229 [45047.331649] [] ? ktime_get_ts64+0x4c/0xf0 [45047.331651] [] ? delayacct_end+0x8f/0xb0 [45047.331653] [] oom_kill_process+0x254/0x3d0 [45047.331655] [] ? oom_unkillable_task+0xcd/0x120 [45047.331657] [] ? find_lock_task_mm+0x56/0xc0 [45047.331659] [] out_of_memory+0x4b6/0x4f0 [45047.331661] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.331662] [] __alloc_pages_nodemask+0x405/0x420 [45047.331664] [] alloc_pages_vma+0xb5/0x200 [45047.331665] [] ? end_swap_bio_write+0x80/0x80 [45047.331667] [] read_swap_cache_async+0xed/0x160 [45047.331669] [] swapin_readahead+0xa8/0x110 [45047.331670] [] handle_mm_fault+0xadb/0xfa0 [45047.331673] [] __do_page_fault+0x154/0x450 [45047.331675] [] do_page_fault+0x35/0x90 [45047.331676] [] page_fault+0x28/0x30 [45047.331677] Mem-Info: [45047.331682] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:71 free_cma:0 [45047.331687] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.331688] lowmem_reserve[]: 0 2580 15620 15620 [45047.331693] Node 0 DMA32 free:59444kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:90 all_unreclaimable? yes [45047.331695] lowmem_reserve[]: 0 0 13040 13040 [45047.331700] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:260kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:291 all_unreclaimable? yes [45047.331701] lowmem_reserve[]: 0 0 0 0 [45047.331705] Node 1 Normal free:45716kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18848kB pagetables:796kB unstable:0kB bounce:0kB free_pcp:24kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1331 all_unreclaimable? yes [45047.331707] lowmem_reserve[]: 0 0 0 0 [45047.331713] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.331721] Node 0 DMA32: 452*4kB (UEM) 719*8kB (UEM) 500*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45047.331728] Node 0 Normal: 172*4kB (UEM) 321*8kB (EM) 576*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36600kB [45047.331733] Node 1 Normal: 1097*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [45047.331734] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.331734] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.331735] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.331735] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.331736] 1792 total pagecache pages [45047.331736] 9 pages in swap cache [45047.331737] Swap cache stats: add 71195, delete 71186, find 15067/20503 [45047.331737] Free swap = 16269348kB [45047.331738] Total swap = 16319484kB [45047.331738] 8369063 pages RAM [45047.331738] 0 pages HighMem/MovableOnly [45047.331739] 241261 pages reserved [45047.331739] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.331746] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.331747] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.331755] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.331756] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.331759] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.331783] [151577] 0 151577 79 0 4 7 0 systemd-cgroups [45047.331784] [151578] 0 151578 79 0 4 13 0 systemd-cgroups [45047.331785] [151582] 0 151582 41 0 2 1 0 systemd-cgroups [45047.331786] [151583] 0 151583 41 0 2 1 0 systemd-cgroups [45047.331787] [151584] 0 151584 41 0 2 1 0 systemd-cgroups [45047.331788] [151585] 0 151585 41 0 2 1 0 systemd-cgroups [45047.331788] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.331789] Out of memory: Kill process 151578 (systemd-cgroups) score 0 or sacrifice child [45047.331790] Killed process 151578 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.337826] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45047.337827] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45047.337828] CPU: 2 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.337829] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.337830] ffff880805c80000 0000000035bd910c ffff880805c8b998 ffffffff816a3e2d [45047.337831] ffff880805c8ba28 ffffffff8169f246 ffffffff810e93ac ffff8803fffb8e50 [45047.337832] ffff8803fffb8e68 0000000000000206 0000000000000206 ffff880805c8ba18 [45047.337832] Call Trace: [45047.337835] [] dump_stack+0x19/0x1b [45047.337836] [] dump_header+0x90/0x229 [45047.337838] [] ? ktime_get_ts64+0x4c/0xf0 [45047.337839] [] ? delayacct_end+0x8f/0xb0 [45047.337841] [] oom_kill_process+0x254/0x3d0 [45047.337842] [] ? oom_unkillable_task+0xcd/0x120 [45047.337844] [] ? find_lock_task_mm+0x56/0xc0 [45047.337845] [] out_of_memory+0x4b6/0x4f0 [45047.337847] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.337848] [] __alloc_pages_nodemask+0x405/0x420 [45047.337850] [] __vmalloc_node_range+0x12f/0x270 [45047.337878] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.337879] [] vzalloc_node+0x47/0x50 [45047.337904] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.337929] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.337957] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45047.337987] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45047.338016] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45047.338017] [] kthread+0xcf/0xe0 [45047.338019] [] ? insert_kthread_work+0x40/0x40 [45047.338020] [] ret_from_fork+0x58/0x90 [45047.338021] [] ? insert_kthread_work+0x40/0x40 [45047.338022] Mem-Info: [45047.338027] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:128 free_cma:0 [45047.338031] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.338033] lowmem_reserve[]: 0 2580 15620 15620 [45047.338037] Node 0 DMA32 free:59444kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:94 all_unreclaimable? yes [45047.338038] lowmem_reserve[]: 0 0 13040 13040 [45047.338043] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:512kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:300 all_unreclaimable? yes [45047.338044] lowmem_reserve[]: 0 0 0 0 [45047.338050] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.338057] Node 0 DMA32: 452*4kB (UEM) 719*8kB (UEM) 500*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45047.338063] Node 0 Normal: 114*4kB (EM) 321*8kB (EM) 575*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36352kB [45047.338064] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.338064] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.338065] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.338066] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.338066] 1792 total pagecache pages [45047.338067] 6 pages in swap cache [45047.338068] Swap cache stats: add 71195, delete 71189, find 15067/20503 [45047.338068] Free swap = 16269368kB [45047.338068] Total swap = 16319484kB [45047.338069] 8369063 pages RAM [45047.338069] 0 pages HighMem/MovableOnly [45047.338070] 241261 pages reserved [45047.338070] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.338078] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.338080] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.338088] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.338089] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.338092] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.338116] [151577] 0 151577 79 0 4 7 0 systemd-cgroups [45047.338117] [151582] 0 151582 41 0 2 1 0 systemd-cgroups [45047.338118] [151583] 0 151583 41 0 2 1 0 systemd-cgroups [45047.338119] [151584] 0 151584 41 0 2 1 0 systemd-cgroups [45047.338120] [151585] 0 151585 41 0 2 1 0 systemd-cgroups [45047.338121] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.338122] Out of memory: Kill process 151577 (systemd-cgroups) score 0 or sacrifice child [45047.338123] Killed process 151577 (systemd-cgroups) total-vm:316kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.341652] systemd-cgroups invoked oom-killer: gfp_mask=0x200da, order=0, oom_score_adj=0 [45047.341653] systemd-cgroups cpuset=/ mems_allowed=0-1 [45047.341655] CPU: 7 PID: 151586 Comm: systemd-cgroups Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.341656] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.341658] ffff8807ce582f70 00000000ab9b0aa1 ffff88004c89b720 ffffffff816a3e2d [45047.341659] ffff88004c89b7b0 ffffffff8169f246 ffffffff810e93ac ffff880362258910 [45047.341661] ffff880362258928 0000000000000202 0000000000000202 ffff88004c89b7a0 [45047.341661] Call Trace: [45047.341665] [] dump_stack+0x19/0x1b [45047.341666] [] dump_header+0x90/0x229 [45047.341668] [] ? ktime_get_ts64+0x4c/0xf0 [45047.341670] [] ? delayacct_end+0x8f/0xb0 [45047.341672] [] oom_kill_process+0x254/0x3d0 [45047.341674] [] ? oom_unkillable_task+0xcd/0x120 [45047.341676] [] ? find_lock_task_mm+0x56/0xc0 [45047.341678] [] out_of_memory+0x4b6/0x4f0 [45047.341679] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.341681] [] __alloc_pages_nodemask+0x405/0x420 [45047.341683] [] alloc_pages_vma+0xb5/0x200 [45047.341685] [] do_cow_fault+0x66/0x290 [45047.341686] [] handle_mm_fault+0x2bd/0xfa0 [45047.341688] [] ? __vma_link_file+0x48/0x70 [45047.341689] [] ? vma_link+0x7d/0xc0 [45047.341692] [] ? vma_set_page_prot+0x3f/0x60 [45047.341694] [] __do_page_fault+0x154/0x450 [45047.341696] [] do_page_fault+0x35/0x90 [45047.341697] [] page_fault+0x28/0x30 [45047.341700] [] ? __clear_user+0x25/0x50 [45047.341702] [] clear_user+0x30/0x40 [45047.341704] [] padzero+0x29/0x40 [45047.341706] [] load_elf_binary+0x8fe/0xe00 [45047.341708] [] ? ima_bprm_check+0x49/0x50 [45047.341710] [] ? load_elf_library+0x220/0x220 [45047.341713] [] search_binary_handler+0xed/0x300 [45047.341714] [] do_execve_common.isra.25+0x5b6/0x6c0 [45047.341716] [] do_execve+0x18/0x20 [45047.341717] [] ____call_usermodehelper+0xfc/0x130 [45047.341718] [] ? ____call_usermodehelper+0x130/0x130 [45047.341719] [] call_helper+0x1e/0x20 [45047.341720] [] ret_from_fork+0x58/0x90 [45047.341721] [] ? ____call_usermodehelper+0x130/0x130 [45047.341722] Mem-Info: [45047.341726] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:216 free_cma:0 [45047.341729] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.341731] lowmem_reserve[]: 0 2580 15620 15620 [45047.341735] Node 0 DMA32 free:59444kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:98 all_unreclaimable? yes [45047.341736] lowmem_reserve[]: 0 0 13040 13040 [45047.341739] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:760kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:309 all_unreclaimable? yes [45047.341740] lowmem_reserve[]: 0 0 0 0 [45047.341743] Node 1 Normal free:45716kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18848kB pagetables:796kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1331 all_unreclaimable? yes [45047.341744] lowmem_reserve[]: 0 0 0 0 [45047.341749] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.341755] Node 0 DMA32: 452*4kB (UEM) 719*8kB (UEM) 500*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45047.341759] Node 0 Normal: 84*4kB (UEM) 321*8kB (EM) 567*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36104kB [45047.341763] Node 1 Normal: 1097*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [45047.341764] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.341764] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.341765] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.341765] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.341766] 1792 total pagecache pages [45047.341766] 9 pages in swap cache [45047.341767] Swap cache stats: add 71199, delete 71190, find 15067/20503 [45047.341767] Free swap = 16269376kB [45047.341767] Total swap = 16319484kB [45047.341768] 8369063 pages RAM [45047.341768] 0 pages HighMem/MovableOnly [45047.341768] 241261 pages reserved [45047.341769] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.341776] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.341777] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.341785] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.341786] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.341789] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.341812] [151582] 0 151582 41 0 2 1 0 systemd-cgroups [45047.341813] [151583] 0 151583 41 0 2 1 0 systemd-cgroups [45047.341814] [151584] 0 151584 41 0 2 1 0 systemd-cgroups [45047.341815] [151585] 0 151585 41 0 2 1 0 systemd-cgroups [45047.341815] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.341816] Out of memory: Kill process 151582 (systemd-cgroups) score 0 or sacrifice child [45047.341817] Killed process 151582 (systemd-cgroups) total-vm:164kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.350031] kworker/u481:18 invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0 [45047.350031] kworker/u481:18 cpuset=/ mems_allowed=0-1 [45047.350033] CPU: 19 PID: 151587 Comm: kworker/u481:18 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.350033] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.350035] ffff8807ce580000 00000000b2b9ec7b ffff880078a6f890 ffffffff816a3e2d [45047.350036] ffff880078a6f920 ffffffff8169f246 ffffffff810e93ac ffff880362258440 [45047.350037] ffff880362258458 0000000000000202 0000000000000202 ffff880078a6f910 [45047.350037] Call Trace: [45047.350040] [] dump_stack+0x19/0x1b [45047.350041] [] dump_header+0x90/0x229 [45047.350042] [] ? ktime_get_ts64+0x4c/0xf0 [45047.350043] [] ? delayacct_end+0x8f/0xb0 [45047.350045] [] oom_kill_process+0x254/0x3d0 [45047.350047] [] ? oom_unkillable_task+0xcd/0x120 [45047.350048] [] ? find_lock_task_mm+0x56/0xc0 [45047.350050] [] out_of_memory+0x4b6/0x4f0 [45047.350052] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.350054] [] __alloc_pages_nodemask+0x405/0x420 [45047.350055] [] alloc_pages_vma+0xb5/0x200 [45047.350057] [] ? anon_vma_interval_tree_insert+0x8b/0x90 [45047.350058] [] handle_mm_fault+0xb60/0xfa0 [45047.350059] [] ? follow_page_mask+0x37/0x780 [45047.350060] [] __get_user_pages+0x17e/0x600 [45047.350061] [] get_user_pages+0x52/0x60 [45047.350063] [] copy_strings.isra.20+0x265/0x340 [45047.350065] [] copy_strings_kernel+0x34/0x40 [45047.350066] [] do_execve_common.isra.25+0x546/0x6c0 [45047.350068] [] ? vfs_rename+0x860/0x930 [45047.350069] [] do_execve+0x18/0x20 [45047.350070] [] ____call_usermodehelper+0xfc/0x130 [45047.350072] [] ? ____call_usermodehelper+0x130/0x130 [45047.350073] [] call_helper+0x1e/0x20 [45047.350074] [] ret_from_fork+0x58/0x90 [45047.350075] [] ? ____call_usermodehelper+0x130/0x130 [45047.350075] Mem-Info: [45047.350079] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:221 free_cma:0 [45047.350083] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.350084] lowmem_reserve[]: 0 2580 15620 15620 [45047.350088] Node 0 DMA32 free:59444kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:102 all_unreclaimable? yes [45047.350089] lowmem_reserve[]: 0 0 13040 13040 [45047.350093] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:772kB local_pcp:244kB free_cma:0kB writeback_tmp:0kB pages_scanned:318 all_unreclaimable? yes [45047.350093] lowmem_reserve[]: 0 0 0 0 [45047.350097] Node 1 Normal free:45716kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18848kB pagetables:796kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:40kB free_cma:0kB writeback_tmp:0kB pages_scanned:1331 all_unreclaimable? yes [45047.350098] lowmem_reserve[]: 0 0 0 0 [45047.350103] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.350108] Node 0 DMA32: 452*4kB (UEM) 719*8kB (UEM) 500*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45047.350112] Node 0 Normal: 84*4kB (UEM) 321*8kB (EM) 567*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36104kB [45047.350117] Node 1 Normal: 1097*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [45047.350117] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.350118] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.350119] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.350119] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.350119] 1792 total pagecache pages [45047.350120] 2 pages in swap cache [45047.350120] Swap cache stats: add 71199, delete 71197, find 15067/20503 [45047.350121] Free swap = 16269388kB [45047.350121] Total swap = 16319484kB [45047.350122] 8369063 pages RAM [45047.350122] 0 pages HighMem/MovableOnly [45047.350122] 241261 pages reserved [45047.350123] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.350130] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.350131] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.350138] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.350140] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.350143] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.350167] [151583] 0 151583 41 0 2 1 0 systemd-cgroups [45047.350168] [151584] 0 151584 41 0 2 1 0 systemd-cgroups [45047.350169] [151585] 0 151585 41 0 2 1 0 systemd-cgroups [45047.350170] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.350171] Out of memory: Kill process 151583 (systemd-cgroups) score 0 or sacrifice child [45047.350172] Killed process 151583 (systemd-cgroups) total-vm:164kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.351169] ll_ost_out01_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45047.351171] ll_ost_out01_00 cpuset=/ mems_allowed=1 [45047.351173] CPU: 31 PID: 32074 Comm: ll_ost_out01_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.351174] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.351177] ffff88080801bf40 000000004282fc3a ffff8803c00ab998 ffffffff816a3e2d [45047.351178] ffff8803c00aba28 ffffffff8169f246 ffffffff810e93ac ffff8804183a1600 [45047.351180] ffff8804183a1618 0000000000000202 0000000000000202 ffff8803c00aba18 [45047.351180] Call Trace: [45047.351184] [] dump_stack+0x19/0x1b [45047.351185] [] dump_header+0x90/0x229 [45047.351187] [] ? ktime_get_ts64+0x4c/0xf0 [45047.351189] [] ? delayacct_end+0x8f/0xb0 [45047.351192] [] oom_kill_process+0x254/0x3d0 [45047.351194] [] ? oom_unkillable_task+0xcd/0x120 [45047.351196] [] ? find_lock_task_mm+0x56/0xc0 [45047.351198] [] out_of_memory+0x4b6/0x4f0 [45047.351200] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.351202] [] __alloc_pages_nodemask+0x405/0x420 [45047.351205] [] __vmalloc_node_range+0x12f/0x270 [45047.351262] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.351264] [] vzalloc_node+0x47/0x50 [45047.351296] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.351329] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.351360] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45047.351393] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45047.351425] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45047.351428] [] kthread+0xcf/0xe0 [45047.351429] [] ? insert_kthread_work+0x40/0x40 [45047.351431] [] ret_from_fork+0x58/0x90 [45047.351433] [] ? insert_kthread_work+0x40/0x40 [45047.351434] Mem-Info: [45047.351439] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:26 free_cma:0 [45047.351445] Node 1 Normal free:45716kB min:45728kB low:57160kB high:68592kB active_anon:0kB inactive_anon:0kB active_file:320kB inactive_file:492kB unevictable:23488kB isolated(anon):0kB isolated(file):0kB present:16777216kB managed:16498508kB mlocked:23488kB dirty:0kB writeback:0kB mapped:1820kB shmem:0kB slab_reclaimable:20496kB slab_unreclaimable:567416kB kernel_stack:18848kB pagetables:796kB unstable:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1331 all_unreclaimable? yes [45047.351447] lowmem_reserve[]: 0 0 0 0 [45047.351455] Node 1 Normal: 1097*4kB (UEM) 728*8kB (UEM) 369*16kB (UEM) 327*32kB (UEM) 269*64kB (UEM) 15*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45716kB [45047.351456] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.351457] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.351458] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.351459] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.351460] 1792 total pagecache pages [45047.351460] 2 pages in swap cache [45047.351461] Swap cache stats: add 71199, delete 71197, find 15067/20503 [45047.351462] Free swap = 16269392kB [45047.351463] Total swap = 16319484kB [45047.351463] 8369063 pages RAM [45047.351464] 0 pages HighMem/MovableOnly [45047.351464] 241261 pages reserved [45047.351465] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.351474] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.351476] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.351485] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.351487] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.351493] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.351521] [151584] 0 151584 41 0 2 1 0 systemd-cgroups [45047.351523] [151585] 0 151585 41 0 2 1 0 systemd-cgroups [45047.351524] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.351526] Out of memory: Kill process 151584 (systemd-cgroups) score 0 or sacrifice child [45047.351527] Killed process 151584 (systemd-cgroups) total-vm:164kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.359414] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45047.359415] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45047.359417] CPU: 3 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.359417] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.359419] ffff880805c80000 0000000035bd910c ffff880805c8b998 ffffffff816a3e2d [45047.359420] ffff880805c8ba28 ffffffff8169f246 ffffffff810e93ac ffff8803fffb8e50 [45047.359421] ffff8803fffb8e68 0000000000000206 0000000000000206 ffff880805c8ba18 [45047.359421] Call Trace: [45047.359423] [] dump_stack+0x19/0x1b [45047.359424] [] dump_header+0x90/0x229 [45047.359426] [] ? ktime_get_ts64+0x4c/0xf0 [45047.359427] [] ? delayacct_end+0x8f/0xb0 [45047.359429] [] oom_kill_process+0x254/0x3d0 [45047.359430] [] ? oom_unkillable_task+0xcd/0x120 [45047.359431] [] ? find_lock_task_mm+0x56/0xc0 [45047.359433] [] out_of_memory+0x4b6/0x4f0 [45047.359434] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.359436] [] __alloc_pages_nodemask+0x405/0x420 [45047.359437] [] __vmalloc_node_range+0x12f/0x270 [45047.359485] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.359487] [] vzalloc_node+0x47/0x50 [45047.359520] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.359551] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.359583] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45047.359618] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45047.359645] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45047.359647] [] kthread+0xcf/0xe0 [45047.359648] [] ? insert_kthread_work+0x40/0x40 [45047.359649] [] ret_from_fork+0x58/0x90 [45047.359650] [] ? insert_kthread_work+0x40/0x40 [45047.359651] Mem-Info: [45047.359654] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:320 free_cma:0 [45047.359658] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.359659] lowmem_reserve[]: 0 2580 15620 15620 [45047.359663] Node 0 DMA32 free:59444kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:110 all_unreclaimable? yes [45047.359664] lowmem_reserve[]: 0 0 13040 13040 [45047.359667] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:1272kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:322 all_unreclaimable? yes [45047.359668] lowmem_reserve[]: 0 0 0 0 [45047.359673] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.359678] Node 0 DMA32: 452*4kB (UEM) 719*8kB (UEM) 500*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45047.359682] Node 0 Normal: 46*4kB (UEM) 310*8kB (UEM) 551*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35608kB [45047.359683] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.359684] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.359684] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.359685] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.359685] 1792 total pagecache pages [45047.359686] 5 pages in swap cache [45047.359686] Swap cache stats: add 71202, delete 71197, find 15067/20504 [45047.359687] Free swap = 16269392kB [45047.359687] Total swap = 16319484kB [45047.359688] 8369063 pages RAM [45047.359688] 0 pages HighMem/MovableOnly [45047.359688] 241261 pages reserved [45047.359688] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.359696] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.359697] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.359705] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.359707] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.359711] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.359736] [151585] 0 151585 41 0 2 1 0 systemd-cgroups [45047.359737] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.359737] Out of memory: Kill process 151585 (systemd-cgroups) score 0 or sacrifice child [45047.359738] Killed process 151585 (systemd-cgroups) total-vm:164kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.369401] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45047.369402] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45047.369404] CPU: 3 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.369404] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.369406] ffff880805c80000 0000000035bd910c ffff880805c8b998 ffffffff816a3e2d [45047.369407] ffff880805c8ba28 ffffffff8169f246 ffffffff810e93ac ffff8803fffb8e50 [45047.369408] ffff8803fffb8e68 0000000000000206 0000000000000206 ffff880805c8ba18 [45047.369408] Call Trace: [45047.369410] [] dump_stack+0x19/0x1b [45047.369411] [] dump_header+0x90/0x229 [45047.369413] [] ? ktime_get_ts64+0x4c/0xf0 [45047.369414] [] ? delayacct_end+0x8f/0xb0 [45047.369416] [] oom_kill_process+0x254/0x3d0 [45047.369417] [] ? oom_unkillable_task+0xcd/0x120 [45047.369418] [] ? find_lock_task_mm+0x56/0xc0 [45047.369420] [] out_of_memory+0x4b6/0x4f0 [45047.369421] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.369423] [] __alloc_pages_nodemask+0x405/0x420 [45047.369424] [] __vmalloc_node_range+0x12f/0x270 [45047.369450] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.369452] [] vzalloc_node+0x47/0x50 [45047.369476] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.369500] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.369523] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45047.369547] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45047.369571] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45047.369573] [] kthread+0xcf/0xe0 [45047.369574] [] ? insert_kthread_work+0x40/0x40 [45047.369575] [] ret_from_fork+0x58/0x90 [45047.369576] [] ? insert_kthread_work+0x40/0x40 [45047.369576] Mem-Info: [45047.369587] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:139 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:300 bounce:0 free:39346 free_pcp:389 free_cma:0 [45047.369592] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.369594] lowmem_reserve[]: 0 2580 15620 15620 [45047.369599] Node 0 DMA32 free:59444kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:80kB unstable:0kB bounce:0kB free_pcp:8kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:122 all_unreclaimable? yes [45047.369600] lowmem_reserve[]: 0 0 13040 13040 [45047.369605] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:1548kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:329 all_unreclaimable? yes [45047.369606] lowmem_reserve[]: 0 0 0 0 [45047.369613] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.369620] Node 0 DMA32: 452*4kB (UEM) 719*8kB (UEM) 500*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59432kB [45047.369627] Node 0 Normal: 44*4kB (E) 294*8kB (EM) 544*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 35360kB [45047.369628] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.369629] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.369629] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.369630] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.369631] 1792 total pagecache pages [45047.369631] 4 pages in swap cache [45047.369632] Swap cache stats: add 71204, delete 71200, find 15067/20505 [45047.369633] Free swap = 16269396kB [45047.369633] Total swap = 16319484kB [45047.369634] 8369063 pages RAM [45047.369634] 0 pages HighMem/MovableOnly [45047.369635] 241261 pages reserved [45047.369635] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.369643] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.369645] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.369654] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.369656] [ 2588] 81 2588 9199 203 21 139 -900 dbus-daemon [45047.369660] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.369688] [151586] 0 151586 41 0 2 1 0 systemd-cgroups [45047.369689] Out of memory: Kill process 151586 (systemd-cgroups) score 0 or sacrifice child [45047.369690] Killed process 151586 (systemd-cgroups) total-vm:164kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [45047.373098] Out of memory: Kill process 2588 (dbus-daemon) score 0 or sacrifice child [45047.373099] Killed process 2588 (dbus-daemon) total-vm:36796kB, anon-rss:0kB, file-rss:812kB, shmem-rss:0kB [45047.450147] ll_ost_out00_00 invoked oom-killer: gfp_mask=0x82d2, order=0, oom_score_adj=0 [45047.450149] ll_ost_out00_00 cpuset=/ mems_allowed=0 [45047.450150] CPU: 3 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.450151] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.450153] ffff880805c80000 0000000035bd910c ffff880805c8ba10 ffffffff816a3e2d [45047.450155] ffff880805c8baa0 ffffffff8169f246 ffffffff8113d71e ffff880805c8ba38 [45047.450156] ffffffff811142f1 ffff880805c8ba78 ffffffff81185e4d ffffffff81185ef6 [45047.450157] Call Trace: [45047.450160] [] dump_stack+0x19/0x1b [45047.450161] [] dump_header+0x90/0x229 [45047.450163] [] ? __delayacct_freepages_end+0x2e/0x30 [45047.450166] [] ? cpuset_mems_allowed_intersects+0x21/0x30 [45047.450168] [] ? oom_unkillable_task+0xcd/0x120 [45047.450170] [] ? find_lock_task_mm+0x56/0xc0 [45047.450172] [] out_of_memory+0x4dc/0x4f0 [45047.450174] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.450176] [] __alloc_pages_nodemask+0x405/0x420 [45047.450177] [] __vmalloc_node_range+0x12f/0x270 [45047.450211] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.450213] [] vzalloc_node+0x47/0x50 [45047.450243] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.450274] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.450303] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45047.450327] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45047.450351] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45047.450353] [] kthread+0xcf/0xe0 [45047.450354] [] ? insert_kthread_work+0x40/0x40 [45047.450355] [] ret_from_fork+0x58/0x90 [45047.450356] [] ? insert_kthread_work+0x40/0x40 [45047.450357] Mem-Info: [45047.450360] active_anon:0 inactive_anon:1 isolated_anon:0 active_file:107 inactive_file:42 isolated_file:0 unevictable:6965 dirty:0 writeback:0 unstable:0 slab_reclaimable:9780 slab_unreclaimable:234736 mapped:1547 shmem:0 pagetables:278 bounce:0 free:39258 free_pcp:0 free_cma:0 [45047.450364] Node 0 DMA free:15576kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15920kB managed:15836kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [45047.450365] lowmem_reserve[]: 0 2580 15620 15620 [45047.450369] Node 0 DMA32 free:59480kB min:7320kB low:9148kB high:10980kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:8kB unevictable:568kB isolated(anon):0kB isolated(file):0kB present:3051628kB managed:2643828kB mlocked:568kB dirty:0kB writeback:0kB mapped:576kB shmem:0kB slab_reclaimable:2236kB slab_unreclaimable:59764kB kernel_stack:1104kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? yes [45047.450370] lowmem_reserve[]: 0 0 13040 13040 [45047.450374] Node 0 Normal free:36648kB min:37012kB low:46264kB high:55516kB active_anon:0kB inactive_anon:4kB active_file:100kB inactive_file:56kB unevictable:3804kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13353036kB mlocked:3804kB dirty:0kB writeback:0kB mapped:3792kB shmem:0kB slab_reclaimable:16388kB slab_unreclaimable:311764kB kernel_stack:10624kB pagetables:324kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:304 all_unreclaimable? yes [45047.450374] lowmem_reserve[]: 0 0 0 0 [45047.450379] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15576kB [45047.450385] Node 0 DMA32: 454*4kB (UEM) 724*8kB (UEM) 502*16kB (UEM) 421*32kB (UEM) 173*64kB (UEM) 25*128kB (UEM) 11*256kB (M) 6*512kB (UM) 4*1024kB (UM) 3*2048kB (M) 0*4096kB = 59512kB [45047.450391] Node 0 Normal: 259*4kB (UEM) 330*8kB (UEM) 577*16kB (UEM) 472*32kB (UEM) 121*64kB (UEM) 10*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37036kB [45047.450392] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.450392] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.450393] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB [45047.450393] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [45047.450394] 1792 total pagecache pages [45047.450394] 1 pages in swap cache [45047.450395] Swap cache stats: add 71213, delete 71212, find 15070/20510 [45047.450395] Free swap = 16269956kB [45047.450396] Total swap = 16319484kB [45047.450396] 8369063 pages RAM [45047.450396] 0 pages HighMem/MovableOnly [45047.450397] 241261 pages reserved [45047.450397] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name [45047.450405] [ 1037] 0 1037 11210 231 24 353 -1000 systemd-udevd [45047.450406] [ 1081] 0 1081 1588718 3975 133 0 -1000 multipathd [45047.450415] [ 2539] 0 2539 13863 112 26 137 -1000 auditd [45047.450419] [ 3695] 0 3695 26499 272 54 247 -1000 sshd [45047.450446] Kernel panic - not syncing: Out of memory and no killable processes... [45047.450448] CPU: 3 PID: 5774 Comm: ll_ost_out00_00 Tainted: P OE ------------ 3.10.0-693.5.2.el7_lustre.x86_64 #1 [45047.450449] Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 [45047.450450] 0000000000000000 0000000035bd910c ffff880805c8ba20 ffffffff816a3e2d [45047.450452] ffff880805c8baa0 ffffffff8169dd14 ffffffff00000008 ffff880805c8bab0 [45047.450453] ffff880805c8ba50 0000000035bd910c 0000000000000110 0000000000000000 [45047.450454] Call Trace: [45047.450456] [] dump_stack+0x19/0x1b [45047.450459] [] panic+0xe8/0x20d [45047.450462] [] out_of_memory+0x4ea/0x4f0 [45047.450463] [] __alloc_pages_slowpath+0x5d6/0x724 [45047.450465] [] __alloc_pages_nodemask+0x405/0x420 [45047.450467] [] __vmalloc_node_range+0x12f/0x270 [45047.450499] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.450501] [] vzalloc_node+0x47/0x50 [45047.450531] [] ? ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.450560] [] ptlrpc_alloc_rqbd+0x1eb/0x590 [ptlrpc] [45047.450590] [] ptlrpc_grow_req_bufs+0xcd/0x290 [ptlrpc] [45047.450620] [] ptlrpc_main+0xdcd/0x1e40 [ptlrpc] [45047.450651] [] ? ptlrpc_register_service+0xe30/0xe30 [ptlrpc] [45047.450652] [] kthread+0xcf/0xe0 [45047.450654] [] ? insert_kthread_work+0x40/0x40 [45047.450655] [] ret_from_fork+0x58/0x90 [45047.450656] [] ? insert_kthread_work+0x40/0x40